Filebeat with 2 kafka outputs












0















I have an issue with Filebeat when I try to send data logs to 2 Kafka nodes at the same time



The following is the Output Kafka section of the filebeat.yml file:



output.kafka:
enabled: true
hosts: [ "192.168.xxx.xx:9092", "192.168.zzz.zz:9092" ]
topic: "syslog"
timeout: 30s
max_message_bytes: 1000000


Both kafka services are running, but only the second node get data. I mean Only the Kafka node 192.168.zzz.zz get the data Filebeat have sent it.



If I exchange the IP adressess, occurs that the second IP address get the data log.



Why occurs that ? What other configurations are need to implement this use case? I need send the data to both kafka outputs.










share|improve this question

























  • Looking at your history of questions, you have accepted none of them. Please refer to What to do when someone answers

    – cricket_007
    Nov 17 '18 at 5:15
















0















I have an issue with Filebeat when I try to send data logs to 2 Kafka nodes at the same time



The following is the Output Kafka section of the filebeat.yml file:



output.kafka:
enabled: true
hosts: [ "192.168.xxx.xx:9092", "192.168.zzz.zz:9092" ]
topic: "syslog"
timeout: 30s
max_message_bytes: 1000000


Both kafka services are running, but only the second node get data. I mean Only the Kafka node 192.168.zzz.zz get the data Filebeat have sent it.



If I exchange the IP adressess, occurs that the second IP address get the data log.



Why occurs that ? What other configurations are need to implement this use case? I need send the data to both kafka outputs.










share|improve this question

























  • Looking at your history of questions, you have accepted none of them. Please refer to What to do when someone answers

    – cricket_007
    Nov 17 '18 at 5:15














0












0








0








I have an issue with Filebeat when I try to send data logs to 2 Kafka nodes at the same time



The following is the Output Kafka section of the filebeat.yml file:



output.kafka:
enabled: true
hosts: [ "192.168.xxx.xx:9092", "192.168.zzz.zz:9092" ]
topic: "syslog"
timeout: 30s
max_message_bytes: 1000000


Both kafka services are running, but only the second node get data. I mean Only the Kafka node 192.168.zzz.zz get the data Filebeat have sent it.



If I exchange the IP adressess, occurs that the second IP address get the data log.



Why occurs that ? What other configurations are need to implement this use case? I need send the data to both kafka outputs.










share|improve this question
















I have an issue with Filebeat when I try to send data logs to 2 Kafka nodes at the same time



The following is the Output Kafka section of the filebeat.yml file:



output.kafka:
enabled: true
hosts: [ "192.168.xxx.xx:9092", "192.168.zzz.zz:9092" ]
topic: "syslog"
timeout: 30s
max_message_bytes: 1000000


Both kafka services are running, but only the second node get data. I mean Only the Kafka node 192.168.zzz.zz get the data Filebeat have sent it.



If I exchange the IP adressess, occurs that the second IP address get the data log.



Why occurs that ? What other configurations are need to implement this use case? I need send the data to both kafka outputs.







apache-kafka filebeat






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 16 '18 at 22:32









cricket_007

81.2k1142111




81.2k1142111










asked Nov 16 '18 at 21:28









Dario RDario R

314




314













  • Looking at your history of questions, you have accepted none of them. Please refer to What to do when someone answers

    – cricket_007
    Nov 17 '18 at 5:15



















  • Looking at your history of questions, you have accepted none of them. Please refer to What to do when someone answers

    – cricket_007
    Nov 17 '18 at 5:15

















Looking at your history of questions, you have accepted none of them. Please refer to What to do when someone answers

– cricket_007
Nov 17 '18 at 5:15





Looking at your history of questions, you have accepted none of them. Please refer to What to do when someone answers

– cricket_007
Nov 17 '18 at 5:15












2 Answers
2






active

oldest

votes


















2














Assuming both broker urls form the same cluster, only one address is used to bootstrap the rest of the cluster. If one of those addresses is not reachable, then the other is selected.



If Filebeat is creating messages with null keys, then messages should be evenly spread across partitions within the specified kafka topic for the cluster that is being connected to.



Data is only sent to the leader for the calculated partition (based on the message key), therefore, a single message cannot be sent to "two nodes (of the same cluster) at the same time". Also, if you had more than those two servers in the Kafka cluster, the one that gets the data could one that is not part of the those addresses you've listed.



I don't think Filebeat can output to more than one unique Kafka cluster at once, at least least not within a single output.kafka section. Logstash might work better for that use case






share|improve this answer

































    0














    As cricket_007 explained, that host array should contain only nodes from the same kafka cluster because they are used to bootstrap your connection to the cluster. The boostraping basically works by providing the address of one, some or all nodes of your cluster so the kafka producer can receive the blueprint (metadata) describing the kafka cluster.



    Furthermore, when you say you can't see your message on one of the nodes, I get the feeling they are not part of the same kafka cluster. If you say you can't see your data because you try to consume from a topic on your "other" server, if they were part of the same cluster you'd be able to consume it anyway even if no partition (leader or replica) are present on that specific node.



    When you consume, your consumer connects to ZK and gets the cluster metadata so it connects to the right nodes/partitions which allow you to consume from a kafka topic, so it doesn't actually depends on the machine where you run your consumer.



    So this answer assumes that you actually want to produce your messages to different kafka clusters.



    In this case, since filebeat doesn't support multiple output blocks of the same output type, you can use one the simplest solution I know for mirroring kafka-to-kafka:
    https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html



    By replicating, you can achieve the same result but instead having your filebeat sending to two kafka clusters, you send only to one and then mirror your topic to a second kafka cluster.






    share|improve this answer























      Your Answer






      StackExchange.ifUsing("editor", function () {
      StackExchange.using("externalEditor", function () {
      StackExchange.using("snippets", function () {
      StackExchange.snippets.init();
      });
      });
      }, "code-snippets");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "1"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: true,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: 10,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53345646%2ffilebeat-with-2-kafka-outputs%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      2














      Assuming both broker urls form the same cluster, only one address is used to bootstrap the rest of the cluster. If one of those addresses is not reachable, then the other is selected.



      If Filebeat is creating messages with null keys, then messages should be evenly spread across partitions within the specified kafka topic for the cluster that is being connected to.



      Data is only sent to the leader for the calculated partition (based on the message key), therefore, a single message cannot be sent to "two nodes (of the same cluster) at the same time". Also, if you had more than those two servers in the Kafka cluster, the one that gets the data could one that is not part of the those addresses you've listed.



      I don't think Filebeat can output to more than one unique Kafka cluster at once, at least least not within a single output.kafka section. Logstash might work better for that use case






      share|improve this answer






























        2














        Assuming both broker urls form the same cluster, only one address is used to bootstrap the rest of the cluster. If one of those addresses is not reachable, then the other is selected.



        If Filebeat is creating messages with null keys, then messages should be evenly spread across partitions within the specified kafka topic for the cluster that is being connected to.



        Data is only sent to the leader for the calculated partition (based on the message key), therefore, a single message cannot be sent to "two nodes (of the same cluster) at the same time". Also, if you had more than those two servers in the Kafka cluster, the one that gets the data could one that is not part of the those addresses you've listed.



        I don't think Filebeat can output to more than one unique Kafka cluster at once, at least least not within a single output.kafka section. Logstash might work better for that use case






        share|improve this answer




























          2












          2








          2







          Assuming both broker urls form the same cluster, only one address is used to bootstrap the rest of the cluster. If one of those addresses is not reachable, then the other is selected.



          If Filebeat is creating messages with null keys, then messages should be evenly spread across partitions within the specified kafka topic for the cluster that is being connected to.



          Data is only sent to the leader for the calculated partition (based on the message key), therefore, a single message cannot be sent to "two nodes (of the same cluster) at the same time". Also, if you had more than those two servers in the Kafka cluster, the one that gets the data could one that is not part of the those addresses you've listed.



          I don't think Filebeat can output to more than one unique Kafka cluster at once, at least least not within a single output.kafka section. Logstash might work better for that use case






          share|improve this answer















          Assuming both broker urls form the same cluster, only one address is used to bootstrap the rest of the cluster. If one of those addresses is not reachable, then the other is selected.



          If Filebeat is creating messages with null keys, then messages should be evenly spread across partitions within the specified kafka topic for the cluster that is being connected to.



          Data is only sent to the leader for the calculated partition (based on the message key), therefore, a single message cannot be sent to "two nodes (of the same cluster) at the same time". Also, if you had more than those two servers in the Kafka cluster, the one that gets the data could one that is not part of the those addresses you've listed.



          I don't think Filebeat can output to more than one unique Kafka cluster at once, at least least not within a single output.kafka section. Logstash might work better for that use case







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Nov 17 '18 at 5:10

























          answered Nov 16 '18 at 22:36









          cricket_007cricket_007

          81.2k1142111




          81.2k1142111

























              0














              As cricket_007 explained, that host array should contain only nodes from the same kafka cluster because they are used to bootstrap your connection to the cluster. The boostraping basically works by providing the address of one, some or all nodes of your cluster so the kafka producer can receive the blueprint (metadata) describing the kafka cluster.



              Furthermore, when you say you can't see your message on one of the nodes, I get the feeling they are not part of the same kafka cluster. If you say you can't see your data because you try to consume from a topic on your "other" server, if they were part of the same cluster you'd be able to consume it anyway even if no partition (leader or replica) are present on that specific node.



              When you consume, your consumer connects to ZK and gets the cluster metadata so it connects to the right nodes/partitions which allow you to consume from a kafka topic, so it doesn't actually depends on the machine where you run your consumer.



              So this answer assumes that you actually want to produce your messages to different kafka clusters.



              In this case, since filebeat doesn't support multiple output blocks of the same output type, you can use one the simplest solution I know for mirroring kafka-to-kafka:
              https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html



              By replicating, you can achieve the same result but instead having your filebeat sending to two kafka clusters, you send only to one and then mirror your topic to a second kafka cluster.






              share|improve this answer




























                0














                As cricket_007 explained, that host array should contain only nodes from the same kafka cluster because they are used to bootstrap your connection to the cluster. The boostraping basically works by providing the address of one, some or all nodes of your cluster so the kafka producer can receive the blueprint (metadata) describing the kafka cluster.



                Furthermore, when you say you can't see your message on one of the nodes, I get the feeling they are not part of the same kafka cluster. If you say you can't see your data because you try to consume from a topic on your "other" server, if they were part of the same cluster you'd be able to consume it anyway even if no partition (leader or replica) are present on that specific node.



                When you consume, your consumer connects to ZK and gets the cluster metadata so it connects to the right nodes/partitions which allow you to consume from a kafka topic, so it doesn't actually depends on the machine where you run your consumer.



                So this answer assumes that you actually want to produce your messages to different kafka clusters.



                In this case, since filebeat doesn't support multiple output blocks of the same output type, you can use one the simplest solution I know for mirroring kafka-to-kafka:
                https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html



                By replicating, you can achieve the same result but instead having your filebeat sending to two kafka clusters, you send only to one and then mirror your topic to a second kafka cluster.






                share|improve this answer


























                  0












                  0








                  0







                  As cricket_007 explained, that host array should contain only nodes from the same kafka cluster because they are used to bootstrap your connection to the cluster. The boostraping basically works by providing the address of one, some or all nodes of your cluster so the kafka producer can receive the blueprint (metadata) describing the kafka cluster.



                  Furthermore, when you say you can't see your message on one of the nodes, I get the feeling they are not part of the same kafka cluster. If you say you can't see your data because you try to consume from a topic on your "other" server, if they were part of the same cluster you'd be able to consume it anyway even if no partition (leader or replica) are present on that specific node.



                  When you consume, your consumer connects to ZK and gets the cluster metadata so it connects to the right nodes/partitions which allow you to consume from a kafka topic, so it doesn't actually depends on the machine where you run your consumer.



                  So this answer assumes that you actually want to produce your messages to different kafka clusters.



                  In this case, since filebeat doesn't support multiple output blocks of the same output type, you can use one the simplest solution I know for mirroring kafka-to-kafka:
                  https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html



                  By replicating, you can achieve the same result but instead having your filebeat sending to two kafka clusters, you send only to one and then mirror your topic to a second kafka cluster.






                  share|improve this answer













                  As cricket_007 explained, that host array should contain only nodes from the same kafka cluster because they are used to bootstrap your connection to the cluster. The boostraping basically works by providing the address of one, some or all nodes of your cluster so the kafka producer can receive the blueprint (metadata) describing the kafka cluster.



                  Furthermore, when you say you can't see your message on one of the nodes, I get the feeling they are not part of the same kafka cluster. If you say you can't see your data because you try to consume from a topic on your "other" server, if they were part of the same cluster you'd be able to consume it anyway even if no partition (leader or replica) are present on that specific node.



                  When you consume, your consumer connects to ZK and gets the cluster metadata so it connects to the right nodes/partitions which allow you to consume from a kafka topic, so it doesn't actually depends on the machine where you run your consumer.



                  So this answer assumes that you actually want to produce your messages to different kafka clusters.



                  In this case, since filebeat doesn't support multiple output blocks of the same output type, you can use one the simplest solution I know for mirroring kafka-to-kafka:
                  https://docs.confluent.io/current/connect/kafka-connect-replicator/index.html



                  By replicating, you can achieve the same result but instead having your filebeat sending to two kafka clusters, you send only to one and then mirror your topic to a second kafka cluster.







                  share|improve this answer












                  share|improve this answer



                  share|improve this answer










                  answered Nov 20 '18 at 15:27









                  Alexandre JumaAlexandre Juma

                  544214




                  544214






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Stack Overflow!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53345646%2ffilebeat-with-2-kafka-outputs%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      這個網誌中的熱門文章

                      Xamarin.form Move up view when keyboard appear

                      Post-Redirect-Get with Spring WebFlux and Thymeleaf

                      Anylogic : not able to use stopDelay()