billiard.pool map function getting hang
I need to run my logic parallelly using all available CPUs, for that I am using billiard package for pooling,
PFB my technology stack -
-Python 3
-Django==2.0.5
-Django rest from work
-celery==4.2.0
My problem is, when I am hitting my rest-API and triggering my celery task it executes till pooling line but after that, its getting hang, I am not getting any error message but It is not moving further.
PFB my project structure:
-cdm
|cmdapp
| |
| |helper
| | |aaa.py
| |tasks
| | |bbb.py
| |api
| |ccc.py
|cdm
| |settings.py
|manage.py
PFB my code example:
#cmdapp.helper.aaa.py
from billiard.pool import Pool
class A():
def squre(self,number):
return number*number
def calculate(self,number_list):
pool = Pool()
result = pool.map(self.squre, number_list)
return result
#cmdapp.tasks.bbb.py
from celery import task
from cmdapp.helper.aaa import A
@task
def calculation_task(number_list):
a_obj=A()
result=a_obj.calculate(number_list)
#cmdapp.api.ccc.py
from rest_framework.views import APIView
from rest_framework import status, permissions
from rest_framework.response import Response
from cmdapp.tasks.bbb import calculation_task
class C(APIView):
def get(self, request):
range_list=range(1000)
calculation_task.delay(range_list)
return Response({
"result": "success",
}, status=status.HTTP_200_OK)
I went through some articles where they have mentioned unpickable function cannot be called by pool.map function. Here I am not getting my function is pickable or unpickable and why It is getting a hang. Please help in that.
python django multithreading celery python-multiprocessing
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
add a comment |
I need to run my logic parallelly using all available CPUs, for that I am using billiard package for pooling,
PFB my technology stack -
-Python 3
-Django==2.0.5
-Django rest from work
-celery==4.2.0
My problem is, when I am hitting my rest-API and triggering my celery task it executes till pooling line but after that, its getting hang, I am not getting any error message but It is not moving further.
PFB my project structure:
-cdm
|cmdapp
| |
| |helper
| | |aaa.py
| |tasks
| | |bbb.py
| |api
| |ccc.py
|cdm
| |settings.py
|manage.py
PFB my code example:
#cmdapp.helper.aaa.py
from billiard.pool import Pool
class A():
def squre(self,number):
return number*number
def calculate(self,number_list):
pool = Pool()
result = pool.map(self.squre, number_list)
return result
#cmdapp.tasks.bbb.py
from celery import task
from cmdapp.helper.aaa import A
@task
def calculation_task(number_list):
a_obj=A()
result=a_obj.calculate(number_list)
#cmdapp.api.ccc.py
from rest_framework.views import APIView
from rest_framework import status, permissions
from rest_framework.response import Response
from cmdapp.tasks.bbb import calculation_task
class C(APIView):
def get(self, request):
range_list=range(1000)
calculation_task.delay(range_list)
return Response({
"result": "success",
}, status=status.HTTP_200_OK)
I went through some articles where they have mentioned unpickable function cannot be called by pool.map function. Here I am not getting my function is pickable or unpickable and why It is getting a hang. Please help in that.
python django multithreading celery python-multiprocessing
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
Is there a reason you're using billiard and not celery? What you are trying to do is much easier if you just use celery yourself.
– 2ps
Nov 23 '18 at 19:06
@2ps, thanks for the time. actually the code that I have posted its just an example. in my real code there is a async job which is calling other function which has complex calculation, thats why I want to use pooling.... just bcz celery not supported multiprocessing I am using billiard.
– Vikram Singh Chandel
Nov 24 '18 at 7:17
add a comment |
I need to run my logic parallelly using all available CPUs, for that I am using billiard package for pooling,
PFB my technology stack -
-Python 3
-Django==2.0.5
-Django rest from work
-celery==4.2.0
My problem is, when I am hitting my rest-API and triggering my celery task it executes till pooling line but after that, its getting hang, I am not getting any error message but It is not moving further.
PFB my project structure:
-cdm
|cmdapp
| |
| |helper
| | |aaa.py
| |tasks
| | |bbb.py
| |api
| |ccc.py
|cdm
| |settings.py
|manage.py
PFB my code example:
#cmdapp.helper.aaa.py
from billiard.pool import Pool
class A():
def squre(self,number):
return number*number
def calculate(self,number_list):
pool = Pool()
result = pool.map(self.squre, number_list)
return result
#cmdapp.tasks.bbb.py
from celery import task
from cmdapp.helper.aaa import A
@task
def calculation_task(number_list):
a_obj=A()
result=a_obj.calculate(number_list)
#cmdapp.api.ccc.py
from rest_framework.views import APIView
from rest_framework import status, permissions
from rest_framework.response import Response
from cmdapp.tasks.bbb import calculation_task
class C(APIView):
def get(self, request):
range_list=range(1000)
calculation_task.delay(range_list)
return Response({
"result": "success",
}, status=status.HTTP_200_OK)
I went through some articles where they have mentioned unpickable function cannot be called by pool.map function. Here I am not getting my function is pickable or unpickable and why It is getting a hang. Please help in that.
python django multithreading celery python-multiprocessing
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
I need to run my logic parallelly using all available CPUs, for that I am using billiard package for pooling,
PFB my technology stack -
-Python 3
-Django==2.0.5
-Django rest from work
-celery==4.2.0
My problem is, when I am hitting my rest-API and triggering my celery task it executes till pooling line but after that, its getting hang, I am not getting any error message but It is not moving further.
PFB my project structure:
-cdm
|cmdapp
| |
| |helper
| | |aaa.py
| |tasks
| | |bbb.py
| |api
| |ccc.py
|cdm
| |settings.py
|manage.py
PFB my code example:
#cmdapp.helper.aaa.py
from billiard.pool import Pool
class A():
def squre(self,number):
return number*number
def calculate(self,number_list):
pool = Pool()
result = pool.map(self.squre, number_list)
return result
#cmdapp.tasks.bbb.py
from celery import task
from cmdapp.helper.aaa import A
@task
def calculation_task(number_list):
a_obj=A()
result=a_obj.calculate(number_list)
#cmdapp.api.ccc.py
from rest_framework.views import APIView
from rest_framework import status, permissions
from rest_framework.response import Response
from cmdapp.tasks.bbb import calculation_task
class C(APIView):
def get(self, request):
range_list=range(1000)
calculation_task.delay(range_list)
return Response({
"result": "success",
}, status=status.HTTP_200_OK)
I went through some articles where they have mentioned unpickable function cannot be called by pool.map function. Here I am not getting my function is pickable or unpickable and why It is getting a hang. Please help in that.
python django multithreading celery python-multiprocessing
python django multithreading celery python-multiprocessing
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
edited Nov 23 '18 at 12:30
Vikram Singh Chandel
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
asked Nov 23 '18 at 11:48
Vikram Singh ChandelVikram Singh Chandel
780924
780924
Is there a reason you're using billiard and not celery? What you are trying to do is much easier if you just use celery yourself.
– 2ps
Nov 23 '18 at 19:06
@2ps, thanks for the time. actually the code that I have posted its just an example. in my real code there is a async job which is calling other function which has complex calculation, thats why I want to use pooling.... just bcz celery not supported multiprocessing I am using billiard.
– Vikram Singh Chandel
Nov 24 '18 at 7:17
add a comment |
Is there a reason you're using billiard and not celery? What you are trying to do is much easier if you just use celery yourself.
– 2ps
Nov 23 '18 at 19:06
@2ps, thanks for the time. actually the code that I have posted its just an example. in my real code there is a async job which is calling other function which has complex calculation, thats why I want to use pooling.... just bcz celery not supported multiprocessing I am using billiard.
– Vikram Singh Chandel
Nov 24 '18 at 7:17
Is there a reason you're using billiard and not celery? What you are trying to do is much easier if you just use celery yourself.
– 2ps
Nov 23 '18 at 19:06
Is there a reason you're using billiard and not celery? What you are trying to do is much easier if you just use celery yourself.
– 2ps
Nov 23 '18 at 19:06
@2ps, thanks for the time. actually the code that I have posted its just an example. in my real code there is a async job which is calling other function which has complex calculation, thats why I want to use pooling.... just bcz celery not supported multiprocessing I am using billiard.
– Vikram Singh Chandel
Nov 24 '18 at 7:17
@2ps, thanks for the time. actually the code that I have posted its just an example. in my real code there is a async job which is calling other function which has complex calculation, thats why I want to use pooling.... just bcz celery not supported multiprocessing I am using billiard.
– Vikram Singh Chandel
Nov 24 '18 at 7:17
add a comment |
0
active
oldest
votes
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53446152%2fbilliard-pool-map-function-getting-hang%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53446152%2fbilliard-pool-map-function-getting-hang%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is there a reason you're using billiard and not celery? What you are trying to do is much easier if you just use celery yourself.
– 2ps
Nov 23 '18 at 19:06
@2ps, thanks for the time. actually the code that I have posted its just an example. in my real code there is a async job which is calling other function which has complex calculation, thats why I want to use pooling.... just bcz celery not supported multiprocessing I am using billiard.
– Vikram Singh Chandel
Nov 24 '18 at 7:17