Yarn of hadoop 3.1.1 can't schedule GPU resource











up vote
1
down vote

favorite
1












I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39















up vote
1
down vote

favorite
1












I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39













up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3




hadoop gpu hadoop3






share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited yesterday









tk421

3,28731326




3,28731326






New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Nov 5 at 2:07









Vic

61




61




New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39


















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39
















Learn how to use code block
– 蕭為元
Nov 5 at 2:15




Learn how to use code block
– 蕭為元
Nov 5 at 2:15












ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39




ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});






Vic is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);

Post as a guest























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes








Vic is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















Vic is a new contributor. Be nice, and check out our Code of Conduct.













Vic is a new contributor. Be nice, and check out our Code of Conduct.












Vic is a new contributor. Be nice, and check out our Code of Conduct.















 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);

Post as a guest






































































-

這個網誌中的熱門文章

Hercules Kyvelos

圖片
Hercules Kyvelos Statistics Real name Hercules Kyvelos Nickname(s) The God Weight(s) Middleweight Light Middleweight Welterweight Height 5 ft 10 in (180 cm) Reach 72 in (183 cm) Nationality Canadian Born ( 1975-02-25 ) February 25, 1975 (age 43) Montreal, Quebec Stance Orthodox Boxing record Total fights 27 Wins 24 Wins by KO 12 Losses 3 Draws 0 No contests 0 Hercules Kyvelos Medal record Men's Boxing Representing   Canada Pan American Games 1995 Mar del Plata Welterweight Hercules Kyvelos (born February 25, 1975) is a Greek-Canadian boxer in the Welterweight division and is the former Canadian Welterweight Champion. [1] He fought at the 1996 Summer Olympics in Atlanta, Georgia. Contents 1 Early life 2 Amateur career 3 Pro career 3.1 WBO Welterweight Championship 3.2 Personal life 4 References 5 External links Early life Hercules was born in
閱讀完整內容

Tangent Lines Diagram Along Smooth Curve

圖片
7 1 I am trying to replicate a diagram and have a minimal example. I don't know how to add additional tangents between the specified points and preserve smoothness. documentclass{article} usepackage{tikz} usetikzlibrary{calc,intersections} begin{document} begin{tikzpicture} % Axes draw [->, name path=x] (-1,0) -- (11,0) node [right] {$x$}; draw [->] (0,-1) -- (0,6) node [above] {$y$}; % Origin node at (0,0) [below left] {$0$}; % Points coordinate (start) at (1,-0.8); coordinate (c1) at (3,3); coordinate (c2) at (5.5,1.5); coordinate (c3) at (8,4); coordinate (end) at (10.5,-0.8); % show the points foreach n in {start,c1,c2,c3,end} fill [black] (n) circle (2pt) node [below] {}; % join the coordinates draw [thick,name path=curve] (start) to[out=70,in=180] (c1) to[out=0,in=180]
閱讀完整內容

Yusuf al-Mu'taman ibn Hud

圖片
Main article: Taifa of Zaragoza Al-Mu'taman Ruler of Zaragoza Reign 1081–1085 Predecessor Ahmad al-Muqtadir Successor Al-Mustain II Born Zaragoza Died 1085 Full name Yusuf ibn Ahmad al-Mu'taman ibn Hūd House Banu Hud Yusuf ibn Ahmad al-Mu'taman ibn Hūd (Arabic: المؤتمن بالله يوسف إبن أحمد إبن هود ‎, al-Mutaman bi l-Lah , died c. 1085) was the third king of the Banu Hud dynasty who ruled over the Taifa of Zaragoza from 1081 to 1085. Yusuf al-Mutaman reigned during the height of power of Muslim Zaragoza , following the thriving period of his father Ahmad al-Muqtadir. He continued his father's efforts and created around him a court of intellectuals, living in the beautiful palace of Aljafería, nicknamed as "the palace of joy". Al-Mu'taman was also a scholarly king and a patron of science, philosophy and arts. As an example of a wise king, he knew astrology, philosophy, and especially mathematics, a discipline in which
閱讀完整內容