Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed
Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.
We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.
We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.
I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.
amazon-web-services amazon-ec2 apache-kafka
asked Nov 14 '18 at 18:21
cnutextcnutext
61
add a comment |
Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.
We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.
We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.
I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.
amazon-web-services amazon-ec2 apache-kafka
asked Nov 14 '18 at 18:21
cnutextcnutext
61
You could just assignbroker.idto a number that you chose. When you move the volume off a failed instance, set the samebroker.idto that server (we also run in AWS and have generation to true, but have set up a value manually)
– cricket_007
Nov 14 '18 at 19:30
thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.
– cnutext
Nov 14 '18 at 20:14
We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.
– cricket_007
Nov 14 '18 at 23:23
Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.
– cnutext
Nov 15 '18 at 12:38
My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id
– cricket_007
Nov 15 '18 at 15:26
add a comment |
Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.
We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.
We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.
I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.
amazon-web-services amazon-ec2 apache-kafka
asked Nov 14 '18 at 18:21
cnutextcnutext
61
Kafka running on AWS - how to retain broker.id on new instance spun-up in-place of instance/broker failed.
We are currently running Kafka in AWS with broker.id gets auto generated. But we are having issues when a broker is failed, new broker/instance spun-up in AWS get assigned with new broker.id. The issue is, with this approach, we need to re-assign the topics/replications on to the new broker manually.
We learned that, replication can be auto resolved by Kafka, if we can manage to get the same broker.id on the new AWS instance spun-up in-place of failed broker/instance.
I have read, we can set broker.id.generation.enable= false, but what is the best way to identify and retain the broker.id? Any links/help is appreciated.
amazon-web-services amazon-ec2 apache-kafka
amazon-web-services amazon-ec2 apache-kafka
asked Nov 14 '18 at 18:21
cnutextcnutext
61
asked Nov 14 '18 at 18:21
cnutextcnutext
61
asked Nov 14 '18 at 18:21
cnutextcnutext
61
asked Nov 14 '18 at 18:21
cnutextcnutext
61
asked Nov 14 '18 at 18:21
cnutextcnutext
61
61
You could just assignbroker.idto a number that you chose. When you move the volume off a failed instance, set the samebroker.idto that server (we also run in AWS and have generation to true, but have set up a value manually)
– cricket_007
Nov 14 '18 at 19:30
thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.
– cnutext
Nov 14 '18 at 20:14
We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.
– cricket_007
Nov 14 '18 at 23:23
Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.
– cnutext
Nov 15 '18 at 12:38
My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id
– cricket_007
Nov 15 '18 at 15:26
add a comment |
You could just assignbroker.idto a number that you chose. When you move the volume off a failed instance, set the samebroker.idto that server (we also run in AWS and have generation to true, but have set up a value manually)
– cricket_007
Nov 14 '18 at 19:30
thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.
– cnutext
Nov 14 '18 at 20:14
We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.
– cricket_007
Nov 14 '18 at 23:23
Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.
– cnutext
Nov 15 '18 at 12:38
My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id
– cricket_007
Nov 15 '18 at 15:26
You could just assign
broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)– cricket_007
Nov 14 '18 at 19:30
You could just assign
broker.id to a number that you chose. When you move the volume off a failed instance, set the same broker.id to that server (we also run in AWS and have generation to true, but have set up a value manually)– cricket_007
Nov 14 '18 at 19:30
thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.
– cnutext
Nov 14 '18 at 20:14
thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.
– cnutext
Nov 14 '18 at 20:14
We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.
– cricket_007
Nov 14 '18 at 23:23
We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.
– cricket_007
Nov 14 '18 at 23:23
Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.
– cnutext
Nov 15 '18 at 12:38
Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.
– cnutext
Nov 15 '18 at 12:38
My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id
– cricket_007
Nov 15 '18 at 15:26
My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id
– cricket_007
Nov 15 '18 at 15:26
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53306532%2fkafka-running-on-aws-how-to-retain-broker-id-on-new-instance-spun-up-in-place%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53306532%2fkafka-running-on-aws-how-to-retain-broker-id-on-new-instance-spun-up-in-place%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You could just assign
broker.idto a number that you chose. When you move the volume off a failed instance, set the samebroker.idto that server (we also run in AWS and have generation to true, but have set up a value manually)– cricket_007
Nov 14 '18 at 19:30
thanks for quick reply. so do you have any custom script which gets triggered during launch of instance to set-up the broker-id manually? something like script mentioned at some script mentioned @ tech.gc.com/scaling-with-kafka ? how do you get previous broker.id which is just failed without conflicting with other/existing broker ids? We don't use EBS, instead we use instance level storage, so we loose storage/volume when AWS instance is terminated.
– cnutext
Nov 14 '18 at 20:14
We use Terraform with a script written in-house, so there isn't any such auto-scaling things. It's all immutable and manually done, but it is using EBS. Our instances don't really drastically fail, though, but if they do, I guess the EBS volume migration is easily done.
– cricket_007
Nov 14 '18 at 23:23
Yeah, EBS is doing trick in your case. Our case is specific: especially with instance level storage and not using EBS. I am sure there should be easy/another way to retain the old broker.id for new AWS instance spun-up for auto replication. Any other ideas/help is appreciated.
– cnutext
Nov 15 '18 at 12:38
My only ideas involve configuration management tools like Ansible and not letting a Kafka service auto-start or pick its own id
– cricket_007
Nov 15 '18 at 15:26