Yarn of hadoop 3.1.1 can't schedule GPU resource

Multi tool use
Multi tool use











up vote
1
down vote

favorite
1












I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39















up vote
1
down vote

favorite
1












I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39













up vote
1
down vote

favorite
1









up vote
1
down vote

favorite
1






1





I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3







share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











I have installed Hadoop 3.1.1 which supports GPU resources. And I configured it according to the official document . However, when I run the word count program with a 800M+ file, the GPU finding resource never are used; at the same time the program works well using memory and vcores(CPU).



All the Prerequisites listed in official document are met: GPU device,CUDA,cgoup etc. GPU works well when I run the docker but Hadoop 3.



Follow are the configuration files:



Resource Type:
enter image description here



capacity-scheduler.xml,other property as default
enter image description here



yarn-site.xml
enter image description here



container-executor.cfg
enter image description here






hadoop gpu hadoop3




hadoop gpu hadoop3






share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited yesterday









tk421

3,28731326




3,28731326






New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked Nov 5 at 2:07









Vic

61




61




New contributor




Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39


















  • Learn how to use code block
    – 蕭為元
    Nov 5 at 2:15










  • ok, I will use it next question. thx.
    – Vic
    Nov 5 at 2:39
















Learn how to use code block
– 蕭為元
Nov 5 at 2:15




Learn how to use code block
– 蕭為元
Nov 5 at 2:15












ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39




ok, I will use it next question. thx.
– Vic
Nov 5 at 2:39

















active

oldest

votes











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});






Vic is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);

Post as a guest























active

oldest

votes













active

oldest

votes









active

oldest

votes






active

oldest

votes








Vic is a new contributor. Be nice, and check out our Code of Conduct.










 

draft saved


draft discarded


















Vic is a new contributor. Be nice, and check out our Code of Conduct.













Vic is a new contributor. Be nice, and check out our Code of Conduct.












Vic is a new contributor. Be nice, and check out our Code of Conduct.















 


draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53147424%2fyarn-of-hadoop-3-1-1-cant-schedule-gpu-resource%23new-answer', 'question_page');
}
);

Post as a guest






































































w8d2wDnd,d1g,s
-
hg340iXSSGO2cnm,39,PN,4,XPSmCqHy2yFzMoMJLsTrywRQ2eCbfDYHN,nm0,Exk XmFDlFQ AGSS1CrpCBQvWsvEn

這個網誌中的熱門文章

Xamarin.form Move up view when keyboard appear

圖片
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I'm trying to build a Chat app UI, the idea of the Layout was pretty simple: When the input bar is focused, keyboard show up and "push" up the chat bar, as it's a grid, the ListView will resize to fit the screen: I update the input bar's margin to "push" it up: NSValue result = (NSValue)args.Notification.UserInfo.ObjectForKey(new NSString(UIKeyboard.FrameEndUserInfoKey)); CGSize keyboardSize = result.RectangleFValue.Size; if (Element != null){ Element.Margin = new Thickness(0, 0, 0,keyboardSize.Height); //push the entry up to keyboard height when keyboard is activated } And this is ...
閱讀完整內容

Post-Redirect-Get with Spring WebFlux and Thymeleaf

圖片
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 2 Can someone explain to me how to implement the post-redirect-get pattern in Spring WebFlux and Thymeleaf? What subscribes on the database save method? @GetMapping("/register") public String showRegisterForm(Model model) { model.addAttribute("user", new User()); return "register"; } @PostMapping public String processRegisterForm(@Valid User user, BindingResult bindingResult) { if (bindingResult.hasErrors()) { return "register"; } else { userRepository.save(user); //what subscribes on this? //how to redirect on e.g. "/login"? ...
閱讀完整內容

Canberra

圖片
capital city of Australia This article is about the capital of Australia. For other uses, see Canberra (disambiguation). City in Australian Capital Territory Canberra Australian Capital Territory Canberra, from top left to bottom right–the city viewed from Mount Ainslie, the Land Axis featuring Old Parliament House and New Parliament House, the Australian War Memorial, the National Carillon, the National Gallery of Australia and the National Library of Australia on Lake Burley Griffin City map plan of Canberra. Canberra Coordinates 35°18′27″S 149°07′27.9″E  /  35.30750°S 149.124417°E  / -35.30750; 149.124417 Coordinates: 35°18′27″S 149°07′27.9″E  /  35.30750°S 149.124417°E  / -35.30750; 149.124417 Population 403,468 (2016) [1]  (8th)  • Density 428.6/km 2  (1,110/sq mi) Established 12 March 1913 Elevation 578 m (1,896 ft) [2] Area 814.2 km 2 (314.4 sq mi) [3] Time zone AEST (UTC+10:00)  • Summer (DST) AEDT (UTC+11...
閱讀完整內容