Yarn of hadoop 3.1.1 can't schedule GPU resource
up vote
1
down vote
favorite
I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).
All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.
Follow are the configuration files:
Resource Type:
capacity-scheduler.xml,other property as default
yarn-site.xml
container-executor.cfg
hadoop gpu hadoop3
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
up vote
1
down vote
favorite
I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).
All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.
Follow are the configuration files:
Resource Type:
capacity-scheduler.xml,other property as default
yarn-site.xml
container-executor.cfg
hadoop gpu hadoop3
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Learn how to use code block
– 蕭為元
Nov 5 at 2:15
ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).
All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.
Follow are the configuration files:
Resource Type:
capacity-scheduler.xml,other property as default
yarn-site.xml
container-executor.cfg
hadoop gpu hadoop3
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).
All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.
Follow are the configuration files:
Resource Type:
capacity-scheduler.xml,other property as default
yarn-site.xml
container-executor.cfg
hadoop gpu hadoop3
hadoop gpu hadoop3
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited yesterday
tk421
3,28731326
3,28731326
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 5 at 2:07
Vic
61
61
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Learn how to use code block
– 蕭為元
Nov 5 at 2:15
ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39
add a comment |
Learn how to use code block
– 蕭為元
Nov 5 at 2:15
ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39
Learn how to use code block
– 蕭為元
Nov 5 at 2:15
Learn how to use code block
– 蕭為元
Nov 5 at 2:15
ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39
ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Vic is a new contributor. Be nice, and check out our Code of Conduct.
Vic is a new contributor. Be nice, and check out our Code of Conduct.
Vic is a new contributor. Be nice, and check out our Code of Conduct.
Vic is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Learn how to use code block
– 蕭為元
Nov 5 at 2:15
ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39