Orchestration: lower CPU usage in co-located containers?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
I am running a distributed data processing experiment using containers managed by Docker Swarm and deployed to azure-based virtual machines.
I am comparing two types of machines: b1ms with 1vcpu and 2gb memory, and b2s with 2vcpu and 4gb memory. According to the documentation, they both use the same Intel haswell i3 processor.
My first question is: does the b2s offer exactly double the processing capacity of the b1ms?
When running my experiments using Flink on a four container cluster deployed to four b1ms machines, the CPU usage was much higher than running the very same experiments on a four container cluster deployed to two b2s machines. The job parallelism was set to four in both cases.
Why does this happen? I would have expected those two setups to be equivalent. The average and max CPU usage for the machines in the 1 container per machine setup is around 30% higher than in the 2 container per machine setup. There is nothing else of any significance running on those machines: Prometheus monitoring alone wouldn't account for such a big discrepancy.
Am I being naïve and overlooking something obvious here?
Maybe the Azure Log Analytics query I am using to get the CPU utilisation per container is inaccurate?
Perf | where ObjectName == "Container" and CounterName == "% Processor Time" | where InstanceName has "taskmanager" | summarize AvgCPUPercent = avg(CounterValue) by Computer, InstanceName // Oql: Type = Perf ObjectName= "Container" CounterName="% Processor Time"| Measure Avg(CounterValue) as AvgCPUPercent by Computer, InstanceName // WorkspaceId: {00000000-0000-0000-0000-000000000000} // Version: 0.1.91
apache-flink cpu-usage docker-swarm
azure-virtual-machine azure-log-analytics asked Nov 24 '18 at 12:07
javalassjavalass
486
add a comment |
I am running a distributed data processing experiment using containers managed by Docker Swarm and deployed to azure-based virtual machines.
I am comparing two types of machines: b1ms with 1vcpu and 2gb memory, and b2s with 2vcpu and 4gb memory. According to the documentation, they both use the same Intel haswell i3 processor.
My first question is: does the b2s offer exactly double the processing capacity of the b1ms?
When running my experiments using Flink on a four container cluster deployed to four b1ms machines, the CPU usage was much higher than running the very same experiments on a four container cluster deployed to two b2s machines. The job parallelism was set to four in both cases.
Why does this happen? I would have expected those two setups to be equivalent. The average and max CPU usage for the machines in the 1 container per machine setup is around 30% higher than in the 2 container per machine setup. There is nothing else of any significance running on those machines: Prometheus monitoring alone wouldn't account for such a big discrepancy.
Am I being naïve and overlooking something obvious here?
Maybe the Azure Log Analytics query I am using to get the CPU utilisation per container is inaccurate?
Perf | where ObjectName == "Container" and CounterName == "% Processor Time" | where InstanceName has "taskmanager" | summarize AvgCPUPercent = avg(CounterValue) by Computer, InstanceName // Oql: Type = Perf ObjectName= "Container" CounterName="% Processor Time"| Measure Avg(CounterValue) as AvgCPUPercent by Computer, InstanceName // WorkspaceId: {00000000-0000-0000-0000-000000000000} // Version: 0.1.91
apache-flink cpu-usage docker-swarm
azure-virtual-machine azure-log-analytics asked Nov 24 '18 at 12:07
javalassjavalass
486
I thought maybe the number of available task slots could be influencing these results, as it was defaulting to the number of cores in the host machine. I changed this setting so that I would have one task slot per task manager. I then repeated the experiments, but the results are the same: clusters where task manager containers are deployed to the same machine seem to be more efficient (i.e. use less CPU) than those where the task managers are deployed to different machines. Why does this happen?
– javalass
Nov 26 '18 at 17:14
add a comment |
I am running a distributed data processing experiment using containers managed by Docker Swarm and deployed to azure-based virtual machines.
I am comparing two types of machines: b1ms with 1vcpu and 2gb memory, and b2s with 2vcpu and 4gb memory. According to the documentation, they both use the same Intel haswell i3 processor.
My first question is: does the b2s offer exactly double the processing capacity of the b1ms?
When running my experiments using Flink on a four container cluster deployed to four b1ms machines, the CPU usage was much higher than running the very same experiments on a four container cluster deployed to two b2s machines. The job parallelism was set to four in both cases.
Why does this happen? I would have expected those two setups to be equivalent. The average and max CPU usage for the machines in the 1 container per machine setup is around 30% higher than in the 2 container per machine setup. There is nothing else of any significance running on those machines: Prometheus monitoring alone wouldn't account for such a big discrepancy.
Am I being naïve and overlooking something obvious here?
Maybe the Azure Log Analytics query I am using to get the CPU utilisation per container is inaccurate?
Perf | where ObjectName == "Container" and CounterName == "% Processor Time" | where InstanceName has "taskmanager" | summarize AvgCPUPercent = avg(CounterValue) by Computer, InstanceName // Oql: Type = Perf ObjectName= "Container" CounterName="% Processor Time"| Measure Avg(CounterValue) as AvgCPUPercent by Computer, InstanceName // WorkspaceId: {00000000-0000-0000-0000-000000000000} // Version: 0.1.91
apache-flink cpu-usage docker-swarm
azure-virtual-machine azure-log-analytics asked Nov 24 '18 at 12:07
javalassjavalass
486
I am running a distributed data processing experiment using containers managed by Docker Swarm and deployed to azure-based virtual machines.
I am comparing two types of machines: b1ms with 1vcpu and 2gb memory, and b2s with 2vcpu and 4gb memory. According to the documentation, they both use the same Intel haswell i3 processor.
My first question is: does the b2s offer exactly double the processing capacity of the b1ms?
When running my experiments using Flink on a four container cluster deployed to four b1ms machines, the CPU usage was much higher than running the very same experiments on a four container cluster deployed to two b2s machines. The job parallelism was set to four in both cases.
Why does this happen? I would have expected those two setups to be equivalent. The average and max CPU usage for the machines in the 1 container per machine setup is around 30% higher than in the 2 container per machine setup. There is nothing else of any significance running on those machines: Prometheus monitoring alone wouldn't account for such a big discrepancy.
Am I being naïve and overlooking something obvious here?
Maybe the Azure Log Analytics query I am using to get the CPU utilisation per container is inaccurate?
Perf | where ObjectName == "Container" and CounterName == "% Processor Time" | where InstanceName has "taskmanager" | summarize AvgCPUPercent = avg(CounterValue) by Computer, InstanceName // Oql: Type = Perf ObjectName= "Container" CounterName="% Processor Time"| Measure Avg(CounterValue) as AvgCPUPercent by Computer, InstanceName // WorkspaceId: {00000000-0000-0000-0000-000000000000} // Version: 0.1.91
apache-flink cpu-usage docker-swarm
azure-virtual-machine azure-log-analytics apache-flink cpu-usage docker-swarm
azure-virtual-machine azure-log-analytics asked Nov 24 '18 at 12:07
javalassjavalass
486
asked Nov 24 '18 at 12:07
javalassjavalass
486
edited Nov 24 '18 at 12:40
javalass
asked Nov 24 '18 at 12:07
javalassjavalass
486
asked Nov 24 '18 at 12:07
javalassjavalass
486
asked Nov 24 '18 at 12:07
javalassjavalass
486
486
I thought maybe the number of available task slots could be influencing these results, as it was defaulting to the number of cores in the host machine. I changed this setting so that I would have one task slot per task manager. I then repeated the experiments, but the results are the same: clusters where task manager containers are deployed to the same machine seem to be more efficient (i.e. use less CPU) than those where the task managers are deployed to different machines. Why does this happen?
– javalass
Nov 26 '18 at 17:14
add a comment |
I thought maybe the number of available task slots could be influencing these results, as it was defaulting to the number of cores in the host machine. I changed this setting so that I would have one task slot per task manager. I then repeated the experiments, but the results are the same: clusters where task manager containers are deployed to the same machine seem to be more efficient (i.e. use less CPU) than those where the task managers are deployed to different machines. Why does this happen?
– javalass
Nov 26 '18 at 17:14
I thought maybe the number of available task slots could be influencing these results, as it was defaulting to the number of cores in the host machine. I changed this setting so that I would have one task slot per task manager. I then repeated the experiments, but the results are the same: clusters where task manager containers are deployed to the same machine seem to be more efficient (i.e. use less CPU) than those where the task managers are deployed to different machines. Why does this happen?
– javalass
Nov 26 '18 at 17:14
I thought maybe the number of available task slots could be influencing these results, as it was defaulting to the number of cores in the host machine. I changed this setting so that I would have one task slot per task manager. I then repeated the experiments, but the results are the same: clusters where task manager containers are deployed to the same machine seem to be more efficient (i.e. use less CPU) than those where the task managers are deployed to different machines. Why does this happen?
– javalass
Nov 26 '18 at 17:14
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53457984%2forchestration-lower-cpu-usage-in-co-located-containers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53457984%2forchestration-lower-cpu-usage-in-co-located-containers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I thought maybe the number of available task slots could be influencing these results, as it was defaulting to the number of cores in the host machine. I changed this setting so that I would have one task slot per task manager. I then repeated the experiments, but the results are the same: clusters where task manager containers are deployed to the same machine seem to be more efficient (i.e. use less CPU) than those where the task managers are deployed to different machines. Why does this happen?
– javalass
Nov 26 '18 at 17:14