What does VIRTUAL_MEMORY_BYTES task counter mean in Hadoop?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
The following excerpt from The Definitive Guide provides high level details as shown below but
- what exactly is virtual memory is referring to in this task counter?
- How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
Following is an example extract from one of the jobs. Physical is 214 GB approx. and virtual is 611 GB approx.
hadoop mapreduce hortonworks-data-platform apache-tez
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
add a comment |
The following excerpt from The Definitive Guide provides high level details as shown below but
- what exactly is virtual memory is referring to in this task counter?
- How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
Following is an example extract from one of the jobs. Physical is 214 GB approx. and virtual is 611 GB approx.
hadoop mapreduce hortonworks-data-platform apache-tez
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
add a comment |
The following excerpt from The Definitive Guide provides high level details as shown below but
- what exactly is virtual memory is referring to in this task counter?
- How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
Following is an example extract from one of the jobs. Physical is 214 GB approx. and virtual is 611 GB approx.
hadoop mapreduce hortonworks-data-platform apache-tez
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
The following excerpt from The Definitive Guide provides high level details as shown below but
- what exactly is virtual memory is referring to in this task counter?
- How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
Following is an example extract from one of the jobs. Physical is 214 GB approx. and virtual is 611 GB approx.
hadoop mapreduce hortonworks-data-platform apache-tez
hadoop mapreduce hortonworks-data-platform apache-tez
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
asked Nov 23 '18 at 18:43
Aravind R. YarramAravind R. Yarram
61.8k35190275
61.8k35190275
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
1.What exactly is virtual memory is referring to in this task counter?
Virtual Memory here is used to prevent Out of Memory errors of a task,if data size doesn't fits in RAM(physical mem).
in RAM.So a portion of memory of size what didn't fit in RAM will be used as Virtual Memory.
So,while setting up hadoop cluster one is advised to have the value of vm.swappiness =1 to achieve better performance. On linux systems, vm.swappiness is set to 60 by default.
Higher the value more aggresive swapping of memory pages.
https://community.hortonworks.com/articles/33522/swappiness-setting-recommendation.html
2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
swapping of memory pages from physical memory to virtual memory on disk when not enough phy mem
This is the relation between PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES.
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53451612%2fwhat-does-virtual-memory-bytes-task-counter-mean-in-hadoop%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
1.What exactly is virtual memory is referring to in this task counter?
Virtual Memory here is used to prevent Out of Memory errors of a task,if data size doesn't fits in RAM(physical mem).
in RAM.So a portion of memory of size what didn't fit in RAM will be used as Virtual Memory.
So,while setting up hadoop cluster one is advised to have the value of vm.swappiness =1 to achieve better performance. On linux systems, vm.swappiness is set to 60 by default.
Higher the value more aggresive swapping of memory pages.
https://community.hortonworks.com/articles/33522/swappiness-setting-recommendation.html
2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
swapping of memory pages from physical memory to virtual memory on disk when not enough phy mem
This is the relation between PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES.
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
add a comment |
1.What exactly is virtual memory is referring to in this task counter?
Virtual Memory here is used to prevent Out of Memory errors of a task,if data size doesn't fits in RAM(physical mem).
in RAM.So a portion of memory of size what didn't fit in RAM will be used as Virtual Memory.
So,while setting up hadoop cluster one is advised to have the value of vm.swappiness =1 to achieve better performance. On linux systems, vm.swappiness is set to 60 by default.
Higher the value more aggresive swapping of memory pages.
https://community.hortonworks.com/articles/33522/swappiness-setting-recommendation.html
2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
swapping of memory pages from physical memory to virtual memory on disk when not enough phy mem
This is the relation between PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES.
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
add a comment |
1.What exactly is virtual memory is referring to in this task counter?
Virtual Memory here is used to prevent Out of Memory errors of a task,if data size doesn't fits in RAM(physical mem).
in RAM.So a portion of memory of size what didn't fit in RAM will be used as Virtual Memory.
So,while setting up hadoop cluster one is advised to have the value of vm.swappiness =1 to achieve better performance. On linux systems, vm.swappiness is set to 60 by default.
Higher the value more aggresive swapping of memory pages.
https://community.hortonworks.com/articles/33522/swappiness-setting-recommendation.html
2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
swapping of memory pages from physical memory to virtual memory on disk when not enough phy mem
This is the relation between PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES.
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
1.What exactly is virtual memory is referring to in this task counter?
Virtual Memory here is used to prevent Out of Memory errors of a task,if data size doesn't fits in RAM(physical mem).
in RAM.So a portion of memory of size what didn't fit in RAM will be used as Virtual Memory.
So,while setting up hadoop cluster one is advised to have the value of vm.swappiness =1 to achieve better performance. On linux systems, vm.swappiness is set to 60 by default.
Higher the value more aggresive swapping of memory pages.
https://community.hortonworks.com/articles/33522/swappiness-setting-recommendation.html
2. How to interpret it? How is it related to PHYSICAL_MEMORY_BYTES?
swapping of memory pages from physical memory to virtual memory on disk when not enough phy mem
This is the relation between PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES.
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
answered Nov 24 '18 at 14:21
Taha NaqviTaha Naqvi
1,134720
1,134720
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
add a comment |
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
How do u think i should interpret the values i have in the question? the vm bytes are way higher than physical mem.
– Aravind R. Yarram
Nov 24 '18 at 15:24
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
vm bytes is sum of ram plus swap space used. The difference between pm vytes and vm bytes is swap space 611 -214 =397 G
– Taha Naqvi
Nov 24 '18 at 18:05
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
Dr Elephant is good option for analysing mr/tez/spark jobs for more clarity..they collect these counters and provide a report github.com/linkedin/dr-elephant/wiki
– Taha Naqvi
Nov 24 '18 at 18:17
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53451612%2fwhat-does-virtual-memory-bytes-task-counter-mean-in-hadoop%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
