How sink topic, kafka to kafka, using Flume?
0
I am trying transfer log from topic to another topic. I need connect Kafka to Kafka using Flume. Take a look below:
#
# Flume Conf
#
a1.sources = s1
a1.sinks = k1
a1.channels = c1
# Kafka Source
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.kafka.bootstrap.servers = kafka:9092
a1.sources.s1.kafka.topics = apache
# Kafka Sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers = kafka:9092
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000000
a1.channels.c1.transactionCapacity = 1000000
# Bind the source and sink to the channel
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1
The sink are not creating.
apache-kafka flume-ng
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
add a comment |
0
I am trying transfer log from topic to another topic. I need connect Kafka to Kafka using Flume. Take a look below:
#
# Flume Conf
#
a1.sources = s1
a1.sinks = k1
a1.channels = c1
# Kafka Source
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.kafka.bootstrap.servers = kafka:9092
a1.sources.s1.kafka.topics = apache
# Kafka Sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers = kafka:9092
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000000
a1.channels.c1.transactionCapacity = 1000000
# Bind the source and sink to the channel
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1
The sink are not creating.
apache-kafka flume-ng
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
1
Why not use MirrorMaker (or other tool built into Kafka)?
– cricket_007
Nov 23 '18 at 2:49
This supports Kerberos? @cricket_007
– Bruno Wego
Nov 23 '18 at 2:50
If you want to continue here, what's the problem/error?? Where's the sink topic configuration?
– cricket_007
Nov 23 '18 at 2:50
1
Yes, MirrorMaker is just a wrapper around a regular producer and consumer. I don't know all the properties for the JAAS stuff and Krb5 JVM settings, but yes it supports it
– cricket_007
Nov 23 '18 at 2:51
I will try this. Thanks a lot @cricket_007
– Bruno Wego
Nov 23 '18 at 2:53
add a comment |
0
0
0
I am trying transfer log from topic to another topic. I need connect Kafka to Kafka using Flume. Take a look below:
#
# Flume Conf
#
a1.sources = s1
a1.sinks = k1
a1.channels = c1
# Kafka Source
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.kafka.bootstrap.servers = kafka:9092
a1.sources.s1.kafka.topics = apache
# Kafka Sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers = kafka:9092
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000000
a1.channels.c1.transactionCapacity = 1000000
# Bind the source and sink to the channel
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1
The sink are not creating.
apache-kafka flume-ng
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
I am trying transfer log from topic to another topic. I need connect Kafka to Kafka using Flume. Take a look below:
#
# Flume Conf
#
a1.sources = s1
a1.sinks = k1
a1.channels = c1
# Kafka Source
a1.sources.s1.type = org.apache.flume.source.kafka.KafkaSource
a1.sources.s1.kafka.bootstrap.servers = kafka:9092
a1.sources.s1.kafka.topics = apache
# Kafka Sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.kafka.bootstrap.servers = kafka:9092
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000000
a1.channels.c1.transactionCapacity = 1000000
# Bind the source and sink to the channel
a1.sources.s1.channels = c1
a1.sinks.k1.channel = c1
The sink are not creating.
apache-kafka flume-ng
apache-kafka flume-ng
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
asked Nov 23 '18 at 2:46
Bruno WegoBruno Wego
2711417
2711417
1
Why not use MirrorMaker (or other tool built into Kafka)?
– cricket_007
Nov 23 '18 at 2:49
This supports Kerberos? @cricket_007
– Bruno Wego
Nov 23 '18 at 2:50
If you want to continue here, what's the problem/error?? Where's the sink topic configuration?
– cricket_007
Nov 23 '18 at 2:50
1
Yes, MirrorMaker is just a wrapper around a regular producer and consumer. I don't know all the properties for the JAAS stuff and Krb5 JVM settings, but yes it supports it
– cricket_007
Nov 23 '18 at 2:51
I will try this. Thanks a lot @cricket_007
– Bruno Wego
Nov 23 '18 at 2:53
add a comment |
1
Why not use MirrorMaker (or other tool built into Kafka)?
– cricket_007
Nov 23 '18 at 2:49
This supports Kerberos? @cricket_007
– Bruno Wego
Nov 23 '18 at 2:50
If you want to continue here, what's the problem/error?? Where's the sink topic configuration?
– cricket_007
Nov 23 '18 at 2:50
1
Yes, MirrorMaker is just a wrapper around a regular producer and consumer. I don't know all the properties for the JAAS stuff and Krb5 JVM settings, but yes it supports it
– cricket_007
Nov 23 '18 at 2:51
I will try this. Thanks a lot @cricket_007
– Bruno Wego
Nov 23 '18 at 2:53
1
1
Why not use MirrorMaker (or other tool built into Kafka)?
– cricket_007
Nov 23 '18 at 2:49
Why not use MirrorMaker (or other tool built into Kafka)?
– cricket_007
Nov 23 '18 at 2:49
This supports Kerberos? @cricket_007
– Bruno Wego
Nov 23 '18 at 2:50
This supports Kerberos? @cricket_007
– Bruno Wego
Nov 23 '18 at 2:50
If you want to continue here, what's the problem/error?? Where's the sink topic configuration?
– cricket_007
Nov 23 '18 at 2:50
If you want to continue here, what's the problem/error?? Where's the sink topic configuration?
– cricket_007
Nov 23 '18 at 2:50
1
1
Yes, MirrorMaker is just a wrapper around a regular producer and consumer. I don't know all the properties for the JAAS stuff and Krb5 JVM settings, but yes it supports it
– cricket_007
Nov 23 '18 at 2:51
Yes, MirrorMaker is just a wrapper around a regular producer and consumer. I don't know all the properties for the JAAS stuff and Krb5 JVM settings, but yes it supports it
– cricket_007
Nov 23 '18 at 2:51
I will try this. Thanks a lot @cricket_007
– Bruno Wego
Nov 23 '18 at 2:53
I will try this. Thanks a lot @cricket_007
– Bruno Wego
Nov 23 '18 at 2:53
add a comment |
1 Answer
1
active
oldest
votes
1
If you want to replicate data from one Kafka cluster to another, there are better ways than Flume, including:
MirrorMaker, as @cricket_007 mentioned (open source, part of Apache Kafka)- Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
If you want a really dirty hack, you can also do something with kafkacat and nc.
Disclaimer: I work for Confluent.
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53440079%2fhow-sink-topic-kafka-to-kafka-using-flume%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1
If you want to replicate data from one Kafka cluster to another, there are better ways than Flume, including:
MirrorMaker, as @cricket_007 mentioned (open source, part of Apache Kafka)- Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
If you want a really dirty hack, you can also do something with kafkacat and nc.
Disclaimer: I work for Confluent.
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
add a comment |
1
If you want to replicate data from one Kafka cluster to another, there are better ways than Flume, including:
MirrorMaker, as @cricket_007 mentioned (open source, part of Apache Kafka)- Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
If you want a really dirty hack, you can also do something with kafkacat and nc.
Disclaimer: I work for Confluent.
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
add a comment |
1
1
1
If you want to replicate data from one Kafka cluster to another, there are better ways than Flume, including:
MirrorMaker, as @cricket_007 mentioned (open source, part of Apache Kafka)- Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
If you want a really dirty hack, you can also do something with kafkacat and nc.
Disclaimer: I work for Confluent.
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
If you want to replicate data from one Kafka cluster to another, there are better ways than Flume, including:
MirrorMaker, as @cricket_007 mentioned (open source, part of Apache Kafka)- Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
If you want a really dirty hack, you can also do something with kafkacat and nc.
Disclaimer: I work for Confluent.
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
answered Nov 23 '18 at 9:13
Robin MoffattRobin Moffatt
7,9431429
7,9431429
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53440079%2fhow-sink-topic-kafka-to-kafka-using-flume%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Why not use MirrorMaker (or other tool built into Kafka)?
– cricket_007
Nov 23 '18 at 2:49
This supports Kerberos? @cricket_007
– Bruno Wego
Nov 23 '18 at 2:50
If you want to continue here, what's the problem/error?? Where's the sink topic configuration?
– cricket_007
Nov 23 '18 at 2:50
1
Yes, MirrorMaker is just a wrapper around a regular producer and consumer. I don't know all the properties for the JAAS stuff and Krb5 JVM settings, but yes it supports it
– cricket_007
Nov 23 '18 at 2:51
I will try this. Thanks a lot @cricket_007
– Bruno Wego
Nov 23 '18 at 2:53