NiFi non-Avro JSON Reader/Writer
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, @Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
json avro apache-nifi
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
add a comment |
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, @Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
json avro apache-nifi
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
add a comment |
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, @Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
json avro apache-nifi
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
It appears that the standard Apache NiFi readers/writers can only parse JSON input based on Avro schema.
Avro schema is limiting for JSON, e.g. it does not allow valid JSON properties starting with digits.
JoltTransformJSON processor can help here (it doesn't impose Avro limitations to how the input JSON may look like), but it seems that this processor does not support batch FlowFiles. It is also not based on the readers and writers (maybe because of that).
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form
{"myprop":"myval","12345":"12345",...}
{"myprop":"myval2","12345":"67890",...}
and transform it to other JSON structure, e.g. defined by JSON schema, and e.g. using JSON Patch transformation, without writing my own processor?
Update
I am using Apache NiFi 1.7.1
Update 2
Unfortunately, @Shu's suggestion did work. I am getting same error.
Reduced the case to a single UpdateRecord processor that reads JSON with numeric properties and writes to a JSON without such properties using
myprop : /data/5836c846e4b0f28d05b40202
mapping. Still same error :(
json avro apache-nifi
json avro apache-nifi
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
edited Nov 22 '18 at 15:50
Sergey Shcherbakov
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
asked Nov 14 '18 at 9:05
Sergey ShcherbakovSergey Shcherbakov
2,17912435
2,17912435
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
1
@SergeyShcherbakov, I don't think we canvalidate field nameswithout using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.
– Shu
Nov 14 '18 at 15:27
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53296423%2fnifi-non-avro-json-reader-writer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
1
@SergeyShcherbakov, I don't think we canvalidate field nameswithout using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.
– Shu
Nov 14 '18 at 15:27
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
add a comment |
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
1
@SergeyShcherbakov, I don't think we canvalidate field nameswithout using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.
– Shu
Nov 14 '18 at 15:27
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
add a comment |
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
it does not allow valid JSON properties starting with digits?
This bug NiFi-4612 fixed in NiFi-1.5 version, We can use AvroSchemaRegistry to defined your schema and change the
Validate Field Names
false
Then we can have avro schema field names starting with digits.
For more details refer to this link.
Is there a way to read arbitrary valid batch JSON input, e.g. in multi-line form?
This bug NiFi-4456 fixed in NiFi-1.7, if you are not using this version of NiFi then we can do workaround to create an array of json messages with ,(comma delimiter) by using.
Flow:
1.SplitText //split the flowfile with 1 line count
2.MergeRecord //merge the flowfiles into one
3.ConvertRecord
For more details regards to this particular issues refer to this link(i have explained with the flow).
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
answered Nov 14 '18 at 14:09
ShuShu
4,6712418
4,6712418
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
1
@SergeyShcherbakov, I don't think we canvalidate field nameswithout using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.
– Shu
Nov 14 '18 at 15:27
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
add a comment |
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
1
@SergeyShcherbakov, I don't think we canvalidate field nameswithout using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.
– Shu
Nov 14 '18 at 15:27
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
Thank you, that's interesting. I am using NiFi 1.7.1. So do you mean that I must use AvroSchemaRegistry to overcome the problem with digits in property names? I was not using the registry before (placing the schema into the processor/controller fields directly. Having solved the porblem with numeric JSON properties would allow me using UpdateRecord (that supports batch processing) instead of JoltTransformJSON processor.
– Sergey Shcherbakov
Nov 14 '18 at 15:17
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
I mean, without the AvroSchemaRegistry it is not possible to change the behaviour e.g. by setting "Validate Field Names" to false? Because I can invalid character exception as soon as property name in my Avro schema starts with a digit in NiFi 1.7.1.
– Sergey Shcherbakov
Nov 14 '18 at 15:19
1
1
@SergeyShcherbakov, I don't think we can
validate field names without using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.– Shu
Nov 14 '18 at 15:27
@SergeyShcherbakov, I don't think we can
validate field names without using AvroSchemaRegistry and i think using UpdateRecord processor to change the filed names will be good way of skipping AvroSchemaRegistry.– Shu
Nov 14 '18 at 15:27
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Thank you! Will give it a try
– Sergey Shcherbakov
Nov 14 '18 at 15:28
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
Please see update in the question. Unfortunately, using AvroSchemaRegistry didn't help :(
– Sergey Shcherbakov
Nov 22 '18 at 15:51
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53296423%2fnifi-non-avro-json-reader-writer%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
