Yarn of hadoop 3.1.1 can't schedule GPU resource











up vote
1
down vote

favorite
1












I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39















up vote
1
down vote

favorite
1












I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39













up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3




hadoop gpu hadoop3






share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited yesterday









tk421

3,28731326




3,28731326






New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Nov 5 at 2:07









Vic

61




61




New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39


















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39
















Learn how to use code block
– 蕭為元
Nov 5 at 2:15




Learn how to use code block
– 蕭為元
Nov 5 at 2:15












ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39




ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});






Vic is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);

Post as a guest























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes








Vic is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















Vic is a new contributor. Be nice, and check out our Code of Conduct.













Vic is a new contributor. Be nice, and check out our Code of Conduct.












Vic is a new contributor. Be nice, and check out our Code of Conduct.















 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);

Post as a guest






































































-

這個網誌中的熱門文章

Xamarin.form Move up view when keyboard appear

圖片
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I'm trying to build a Chat app UI, the idea of the Layout was pretty simple: When the input bar is focused, keyboard show up and "push" up the chat bar, as it's a grid, the ListView will resize to fit the screen: I update the input bar's margin to "push" it up: NSValue result = (NSValue)args.Notification.UserInfo.ObjectForKey(new NSString(UIKeyboard.FrameEndUserInfoKey)); CGSize keyboardSize = result.RectangleFValue.Size; if (Element != null){ Element.Margin = new Thickness(0, 0, 0,keyboardSize.Height); //push the entry up to keyboard height when keyboard is activated } And this is ...
閱讀完整內容

Post-Redirect-Get with Spring WebFlux and Thymeleaf

圖片
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 2 Can someone explain to me how to implement the post-redirect-get pattern in Spring WebFlux and Thymeleaf? What subscribes on the database save method? @GetMapping("/register") public String showRegisterForm(Model model) { model.addAttribute("user", new User()); return "register"; } @PostMapping public String processRegisterForm(@Valid User user, BindingResult bindingResult) { if (bindingResult.hasErrors()) { return "register"; } else { userRepository.save(user); //what subscribes on this? //how to redirect on e.g. "/login"? ...
閱讀完整內容

Anylogic : not able to use stopDelay()

圖片
-2 I am building a railway station model. I want a train to depart when I click a button. I want to call stopDelay() for that but I am not sure how to do that and from which place. anylogic share | improve this question asked Nov 23 '18 at 10:12 Anupam Singh Anupam Singh 1 Hi Anupam, welcome to SOF. Please check how to ask good questions, see link below. Your question is...
閱讀完整內容