Distributing jobs evenly across multiple GPUs with `multiprocessing.Pool`












3















Let's say that I have the following:




  • A system with 4 GPUs.

  • A function, foo, which may be run up to 2 times simultaneously on each GPU.

  • A list of files that need to be processed using foo in any order. However, each file takes an unpredictable amount of time to be processed.


I would like to process all the files, keeping all the GPUs as busy as possible by ensuring there are always 8 instances of foo running at any given time (2 instance on each GPU) until less than 8 files remain.



The actual details of invoking the GPU are not my issue. What I'm trying to figure out is how to write the parallelization so that I can keep 8 instances of foo running but somehow making sure that exactly 2 of each GPU ID are used at all times.



I've come up with one way to solve this problem using multiprocessing.Pool, but the solution is quite brittle and relies on (AFAIK) undocumented features. It relies on the fact that the processes within the Pool are named in the format FormPoolWorker-%d where %d is a number between one and the number of processes in the pool. I take this value and mod it with the number of GPUs and that gives me a valid GPU id. However, it would be much nicer if I could somehow give the GPU id directly to each process, perhaps on initialization, instead of relying on the string format of the process names.



One thing I considered is that if the initializer and initargs parameters of Pool.__init__ allowed for a list of initargs so that each process could be initialized with a different set of arguments then the problem would be moot. Unfortunately that doesn't appear to work.



Can anybody recommend a more robust or pythonic solution to this problem?



Hacky solution (Python 3.7):



from multiprocessing import Pool, current_process

def foo(filename):
# Hacky way to get a GPU id using process name (format "ForkPoolWorker-%d")
gpu_id = (int(current_process().name.split('-')[-1]) - 1) % 4

# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))

pool = Pool(processes=4*2)

files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()









share|improve this question




















  • 1





    Where does GPU come into play? Your example uses CPUs.

    – Darkonaut
    Nov 22 '18 at 10:52






  • 1





    The GPU processing would be invoked in the comment # ... process filename. I edited the post to make that more clear.

    – jodag
    Nov 22 '18 at 16:18


















3















Let's say that I have the following:




  • A system with 4 GPUs.

  • A function, foo, which may be run up to 2 times simultaneously on each GPU.

  • A list of files that need to be processed using foo in any order. However, each file takes an unpredictable amount of time to be processed.


I would like to process all the files, keeping all the GPUs as busy as possible by ensuring there are always 8 instances of foo running at any given time (2 instance on each GPU) until less than 8 files remain.



The actual details of invoking the GPU are not my issue. What I'm trying to figure out is how to write the parallelization so that I can keep 8 instances of foo running but somehow making sure that exactly 2 of each GPU ID are used at all times.



I've come up with one way to solve this problem using multiprocessing.Pool, but the solution is quite brittle and relies on (AFAIK) undocumented features. It relies on the fact that the processes within the Pool are named in the format FormPoolWorker-%d where %d is a number between one and the number of processes in the pool. I take this value and mod it with the number of GPUs and that gives me a valid GPU id. However, it would be much nicer if I could somehow give the GPU id directly to each process, perhaps on initialization, instead of relying on the string format of the process names.



One thing I considered is that if the initializer and initargs parameters of Pool.__init__ allowed for a list of initargs so that each process could be initialized with a different set of arguments then the problem would be moot. Unfortunately that doesn't appear to work.



Can anybody recommend a more robust or pythonic solution to this problem?



Hacky solution (Python 3.7):



from multiprocessing import Pool, current_process

def foo(filename):
# Hacky way to get a GPU id using process name (format "ForkPoolWorker-%d")
gpu_id = (int(current_process().name.split('-')[-1]) - 1) % 4

# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))

pool = Pool(processes=4*2)

files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()









share|improve this question




















  • 1





    Where does GPU come into play? Your example uses CPUs.

    – Darkonaut
    Nov 22 '18 at 10:52






  • 1





    The GPU processing would be invoked in the comment # ... process filename. I edited the post to make that more clear.

    – jodag
    Nov 22 '18 at 16:18
















3












3








3


2






Let's say that I have the following:




  • A system with 4 GPUs.

  • A function, foo, which may be run up to 2 times simultaneously on each GPU.

  • A list of files that need to be processed using foo in any order. However, each file takes an unpredictable amount of time to be processed.


I would like to process all the files, keeping all the GPUs as busy as possible by ensuring there are always 8 instances of foo running at any given time (2 instance on each GPU) until less than 8 files remain.



The actual details of invoking the GPU are not my issue. What I'm trying to figure out is how to write the parallelization so that I can keep 8 instances of foo running but somehow making sure that exactly 2 of each GPU ID are used at all times.



I've come up with one way to solve this problem using multiprocessing.Pool, but the solution is quite brittle and relies on (AFAIK) undocumented features. It relies on the fact that the processes within the Pool are named in the format FormPoolWorker-%d where %d is a number between one and the number of processes in the pool. I take this value and mod it with the number of GPUs and that gives me a valid GPU id. However, it would be much nicer if I could somehow give the GPU id directly to each process, perhaps on initialization, instead of relying on the string format of the process names.



One thing I considered is that if the initializer and initargs parameters of Pool.__init__ allowed for a list of initargs so that each process could be initialized with a different set of arguments then the problem would be moot. Unfortunately that doesn't appear to work.



Can anybody recommend a more robust or pythonic solution to this problem?



Hacky solution (Python 3.7):



from multiprocessing import Pool, current_process

def foo(filename):
# Hacky way to get a GPU id using process name (format "ForkPoolWorker-%d")
gpu_id = (int(current_process().name.split('-')[-1]) - 1) % 4

# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))

pool = Pool(processes=4*2)

files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()









share|improve this question
















Let's say that I have the following:




  • A system with 4 GPUs.

  • A function, foo, which may be run up to 2 times simultaneously on each GPU.

  • A list of files that need to be processed using foo in any order. However, each file takes an unpredictable amount of time to be processed.


I would like to process all the files, keeping all the GPUs as busy as possible by ensuring there are always 8 instances of foo running at any given time (2 instance on each GPU) until less than 8 files remain.



The actual details of invoking the GPU are not my issue. What I'm trying to figure out is how to write the parallelization so that I can keep 8 instances of foo running but somehow making sure that exactly 2 of each GPU ID are used at all times.



I've come up with one way to solve this problem using multiprocessing.Pool, but the solution is quite brittle and relies on (AFAIK) undocumented features. It relies on the fact that the processes within the Pool are named in the format FormPoolWorker-%d where %d is a number between one and the number of processes in the pool. I take this value and mod it with the number of GPUs and that gives me a valid GPU id. However, it would be much nicer if I could somehow give the GPU id directly to each process, perhaps on initialization, instead of relying on the string format of the process names.



One thing I considered is that if the initializer and initargs parameters of Pool.__init__ allowed for a list of initargs so that each process could be initialized with a different set of arguments then the problem would be moot. Unfortunately that doesn't appear to work.



Can anybody recommend a more robust or pythonic solution to this problem?



Hacky solution (Python 3.7):



from multiprocessing import Pool, current_process

def foo(filename):
# Hacky way to get a GPU id using process name (format "ForkPoolWorker-%d")
gpu_id = (int(current_process().name.split('-')[-1]) - 1) % 4

# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))

pool = Pool(processes=4*2)

files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()






python python-multiprocessing






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 22 '18 at 16:23







jodag

















asked Nov 22 '18 at 1:39









jodagjodag

4,62821330




4,62821330








  • 1





    Where does GPU come into play? Your example uses CPUs.

    – Darkonaut
    Nov 22 '18 at 10:52






  • 1





    The GPU processing would be invoked in the comment # ... process filename. I edited the post to make that more clear.

    – jodag
    Nov 22 '18 at 16:18
















  • 1





    Where does GPU come into play? Your example uses CPUs.

    – Darkonaut
    Nov 22 '18 at 10:52






  • 1





    The GPU processing would be invoked in the comment # ... process filename. I edited the post to make that more clear.

    – jodag
    Nov 22 '18 at 16:18










1




1





Where does GPU come into play? Your example uses CPUs.

– Darkonaut
Nov 22 '18 at 10:52





Where does GPU come into play? Your example uses CPUs.

– Darkonaut
Nov 22 '18 at 10:52




1




1





The GPU processing would be invoked in the comment # ... process filename. I edited the post to make that more clear.

– jodag
Nov 22 '18 at 16:18







The GPU processing would be invoked in the comment # ... process filename. I edited the post to make that more clear.

– jodag
Nov 22 '18 at 16:18














1 Answer
1






active

oldest

votes


















3














I figured it out. It's actually quite simple. All we need to do is use a multiprocessing.Queue to manage the available GPU IDs. Start by initializing the Queue to contain 2 of each GPU ID, then get the GPU ID from the queue at the beginning of foo and put it back at the end.



from multiprocessing import Pool, current_process, Queue

NUM_GPUS = 4
PROC_PER_GPU = 2

queue = Queue()

def foo(filename):
gpu_id = queue.get()
try:
# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))
finally:
queue.put(gpu_id)

# initialize the queue with the GPU ids
for gpu_ids in range(NUM_GPUS):
for _ in range(PROC_PER_GPU):
queue.put(gpu_ids)

pool = Pool(processes=PROC_PER_GPU * NUM_GPUS)
files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()





share|improve this answer





















  • 2





    Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

    – dwagnerkc
    Nov 22 '18 at 4:45











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53422761%2fdistributing-jobs-evenly-across-multiple-gpus-with-multiprocessing-pool%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









3














I figured it out. It's actually quite simple. All we need to do is use a multiprocessing.Queue to manage the available GPU IDs. Start by initializing the Queue to contain 2 of each GPU ID, then get the GPU ID from the queue at the beginning of foo and put it back at the end.



from multiprocessing import Pool, current_process, Queue

NUM_GPUS = 4
PROC_PER_GPU = 2

queue = Queue()

def foo(filename):
gpu_id = queue.get()
try:
# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))
finally:
queue.put(gpu_id)

# initialize the queue with the GPU ids
for gpu_ids in range(NUM_GPUS):
for _ in range(PROC_PER_GPU):
queue.put(gpu_ids)

pool = Pool(processes=PROC_PER_GPU * NUM_GPUS)
files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()





share|improve this answer





















  • 2





    Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

    – dwagnerkc
    Nov 22 '18 at 4:45
















3














I figured it out. It's actually quite simple. All we need to do is use a multiprocessing.Queue to manage the available GPU IDs. Start by initializing the Queue to contain 2 of each GPU ID, then get the GPU ID from the queue at the beginning of foo and put it back at the end.



from multiprocessing import Pool, current_process, Queue

NUM_GPUS = 4
PROC_PER_GPU = 2

queue = Queue()

def foo(filename):
gpu_id = queue.get()
try:
# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))
finally:
queue.put(gpu_id)

# initialize the queue with the GPU ids
for gpu_ids in range(NUM_GPUS):
for _ in range(PROC_PER_GPU):
queue.put(gpu_ids)

pool = Pool(processes=PROC_PER_GPU * NUM_GPUS)
files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()





share|improve this answer





















  • 2





    Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

    – dwagnerkc
    Nov 22 '18 at 4:45














3












3








3







I figured it out. It's actually quite simple. All we need to do is use a multiprocessing.Queue to manage the available GPU IDs. Start by initializing the Queue to contain 2 of each GPU ID, then get the GPU ID from the queue at the beginning of foo and put it back at the end.



from multiprocessing import Pool, current_process, Queue

NUM_GPUS = 4
PROC_PER_GPU = 2

queue = Queue()

def foo(filename):
gpu_id = queue.get()
try:
# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))
finally:
queue.put(gpu_id)

# initialize the queue with the GPU ids
for gpu_ids in range(NUM_GPUS):
for _ in range(PROC_PER_GPU):
queue.put(gpu_ids)

pool = Pool(processes=PROC_PER_GPU * NUM_GPUS)
files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()





share|improve this answer















I figured it out. It's actually quite simple. All we need to do is use a multiprocessing.Queue to manage the available GPU IDs. Start by initializing the Queue to contain 2 of each GPU ID, then get the GPU ID from the queue at the beginning of foo and put it back at the end.



from multiprocessing import Pool, current_process, Queue

NUM_GPUS = 4
PROC_PER_GPU = 2

queue = Queue()

def foo(filename):
gpu_id = queue.get()
try:
# run processing on GPU <gpu_id>
ident = current_process().ident
print('{}: starting process on GPU {}'.format(ident, gpu_id))
# ... process filename
print('{}: finished'.format(ident))
finally:
queue.put(gpu_id)

# initialize the queue with the GPU ids
for gpu_ids in range(NUM_GPUS):
for _ in range(PROC_PER_GPU):
queue.put(gpu_ids)

pool = Pool(processes=PROC_PER_GPU * NUM_GPUS)
files = ['file{}.xyz'.format(x) for x in range(1000)]
for _ in pool.imap_unordered(foo, files):
pass
pool.close()
pool.join()






share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 24 '18 at 0:36

























answered Nov 22 '18 at 4:31









jodagjodag

4,62821330




4,62821330








  • 2





    Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

    – dwagnerkc
    Nov 22 '18 at 4:45














  • 2





    Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

    – dwagnerkc
    Nov 22 '18 at 4:45








2




2





Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

– dwagnerkc
Nov 22 '18 at 4:45





Very nice solution. I spent some time on your question and determined that I liked your original hack! I'm sure it will come in handy someday.

– dwagnerkc
Nov 22 '18 at 4:45




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53422761%2fdistributing-jobs-evenly-across-multiple-gpus-with-multiprocessing-pool%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







這個網誌中的熱門文章

Xamarin.form Move up view when keyboard appear

Post-Redirect-Get with Spring WebFlux and Thymeleaf

Anylogic : not able to use stopDelay()