Kafka Streams: How to force quicker updates on stores












0















I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.



In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.



How can I reduce this waiting time so computations are triggered faster in my tests?



Which settings are involved?



I tried setting commit.interval.ms to 100 but it is of no help.










share|improve this question























  • Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.

    – Matthias J. Sax
    Nov 21 '18 at 17:52











  • I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the .toStream() - where they weren't before. I'll investigate, but any hint on this?

    – xmar
    Nov 22 '18 at 11:27











  • Not sure exactly what computation you do. But <key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?

    – Matthias J. Sax
    Nov 23 '18 at 5:11
















0















I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.



In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.



How can I reduce this waiting time so computations are triggered faster in my tests?



Which settings are involved?



I tried setting commit.interval.ms to 100 but it is of no help.










share|improve this question























  • Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.

    – Matthias J. Sax
    Nov 21 '18 at 17:52











  • I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the .toStream() - where they weren't before. I'll investigate, but any hint on this?

    – xmar
    Nov 22 '18 at 11:27











  • Not sure exactly what computation you do. But <key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?

    – Matthias J. Sax
    Nov 23 '18 at 5:11














0












0








0








I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.



In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.



How can I reduce this waiting time so computations are triggered faster in my tests?



Which settings are involved?



I tried setting commit.interval.ms to 100 but it is of no help.










share|improve this question














I have an aggregated store via DSL via windowed aggregation over a stream.
Then, I get a stream from it via toStream, do some small transformation and send to another topic.



In my tests, from the publishing time to the time when the data is in the destination stream, it may well spend 10-20 seconds. Which makes my tests having to wait for it that long.



How can I reduce this waiting time so computations are triggered faster in my tests?



Which settings are involved?



I tried setting commit.interval.ms to 100 but it is of no help.







apache-kafka apache-kafka-streams






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 21 '18 at 15:18









xmarxmar

475214




475214













  • Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.

    – Matthias J. Sax
    Nov 21 '18 at 17:52











  • I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the .toStream() - where they weren't before. I'll investigate, but any hint on this?

    – xmar
    Nov 22 '18 at 11:27











  • Not sure exactly what computation you do. But <key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?

    – Matthias J. Sax
    Nov 23 '18 at 5:11



















  • Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.

    – Matthias J. Sax
    Nov 21 '18 at 17:52











  • I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the .toStream() - where they weren't before. I'll investigate, but any hint on this?

    – xmar
    Nov 22 '18 at 11:27











  • Not sure exactly what computation you do. But <key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?

    – Matthias J. Sax
    Nov 23 '18 at 5:11

















Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.

– Matthias J. Sax
Nov 21 '18 at 17:52





Reducing the commit interval should actually work... You can also try to disable caching by setting the cache size to zero in the config. Hope this helps.

– Matthias J. Sax
Nov 21 '18 at 17:52













I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the .toStream() - where they weren't before. I'll investigate, but any hint on this?

– xmar
Nov 22 '18 at 11:27





I tried setting cache to zero. However, I see messages coming as null downstream in the topoogy in stages after the .toStream() - where they weren't before. I'll investigate, but any hint on this?

– xmar
Nov 22 '18 at 11:27













Not sure exactly what computation you do. But <key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?

– Matthias J. Sax
Nov 23 '18 at 5:11





Not sure exactly what computation you do. But <key,null> messages indicate tombstones (ie, delete on the KTable). With caching, those might get deduplicate?

– Matthias J. Sax
Nov 23 '18 at 5:11












0






active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53415184%2fkafka-streams-how-to-force-quicker-updates-on-stores%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes
















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53415184%2fkafka-streams-how-to-force-quicker-updates-on-stores%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







這個網誌中的熱門文章

Xamarin.form Move up view when keyboard appear

Post-Redirect-Get with Spring WebFlux and Thymeleaf

Anylogic : not able to use stopDelay()