How would I run a queue of an Amazon SQS ? Is it a cron job?
up vote
0
down vote
favorite
I understand the basic of Amazon SQS. Yet i'm still confused on how it runs? Is it an infinite running function that polls messages and deal with it? how would I achieve that in php?
What I have in mind is a cron job that triggers the polling and process the messages. is my understanding right?
php amazon-web-services amazon-sqs
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
asked Nov 9 at 17:30
Faisal
3451417
add a comment |
up vote
0
down vote
favorite
I understand the basic of Amazon SQS. Yet i'm still confused on how it runs? Is it an infinite running function that polls messages and deal with it? how would I achieve that in php?
What I have in mind is a cron job that triggers the polling and process the messages. is my understanding right?
php amazon-web-services amazon-sqs
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
asked Nov 9 at 17:30
Faisal
3451417
1
I don't think that PHP could be "infinite running"
– GrumpyCrouton
Nov 9 at 17:34
Use a process manager like supervisord to keep one or more worker scripts running, and in your worker keep track of how many jobs you've processed and shut down after 1000 or so to make sure PHP's resource usage doesn't go nuts. Supervisord will make sure that there are always workers running.
– Sammitch
Nov 9 at 17:38
1
@GrumpyCrouton PHP can absolutely be infinite running, if run via the CLI.
– ceejayoz
Nov 9 at 17:55
@ceejayoz Interesting, thank you.
– GrumpyCrouton
Nov 9 at 18:37
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I understand the basic of Amazon SQS. Yet i'm still confused on how it runs? Is it an infinite running function that polls messages and deal with it? how would I achieve that in php?
What I have in mind is a cron job that triggers the polling and process the messages. is my understanding right?
php amazon-web-services amazon-sqs
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
asked Nov 9 at 17:30
Faisal
3451417
I understand the basic of Amazon SQS. Yet i'm still confused on how it runs? Is it an infinite running function that polls messages and deal with it? how would I achieve that in php?
What I have in mind is a cron job that triggers the polling and process the messages. is my understanding right?
php amazon-web-services amazon-sqs
php amazon-web-services amazon-sqs
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
asked Nov 9 at 17:30
Faisal
3451417
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
asked Nov 9 at 17:30
Faisal
3451417
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
edited Nov 11 at 4:56
John Rotenstein
66.5k774118
66.5k774118
asked Nov 9 at 17:30
Faisal
3451417
asked Nov 9 at 17:30
Faisal
3451417
asked Nov 9 at 17:30
Faisal
3451417
3451417
1
I don't think that PHP could be "infinite running"
– GrumpyCrouton
Nov 9 at 17:34
Use a process manager like supervisord to keep one or more worker scripts running, and in your worker keep track of how many jobs you've processed and shut down after 1000 or so to make sure PHP's resource usage doesn't go nuts. Supervisord will make sure that there are always workers running.
– Sammitch
Nov 9 at 17:38
1
@GrumpyCrouton PHP can absolutely be infinite running, if run via the CLI.
– ceejayoz
Nov 9 at 17:55
@ceejayoz Interesting, thank you.
– GrumpyCrouton
Nov 9 at 18:37
add a comment |
1
I don't think that PHP could be "infinite running"
– GrumpyCrouton
Nov 9 at 17:34
Use a process manager like supervisord to keep one or more worker scripts running, and in your worker keep track of how many jobs you've processed and shut down after 1000 or so to make sure PHP's resource usage doesn't go nuts. Supervisord will make sure that there are always workers running.
– Sammitch
Nov 9 at 17:38
1
@GrumpyCrouton PHP can absolutely be infinite running, if run via the CLI.
– ceejayoz
Nov 9 at 17:55
@ceejayoz Interesting, thank you.
– GrumpyCrouton
Nov 9 at 18:37
1
1
I don't think that PHP could be "infinite running"
– GrumpyCrouton
Nov 9 at 17:34
I don't think that PHP could be "infinite running"
– GrumpyCrouton
Nov 9 at 17:34
Use a process manager like supervisord to keep one or more worker scripts running, and in your worker keep track of how many jobs you've processed and shut down after 1000 or so to make sure PHP's resource usage doesn't go nuts. Supervisord will make sure that there are always workers running.
– Sammitch
Nov 9 at 17:38
Use a process manager like supervisord to keep one or more worker scripts running, and in your worker keep track of how many jobs you've processed and shut down after 1000 or so to make sure PHP's resource usage doesn't go nuts. Supervisord will make sure that there are always workers running.
– Sammitch
Nov 9 at 17:38
1
1
@GrumpyCrouton PHP can absolutely be infinite running, if run via the CLI.
– ceejayoz
Nov 9 at 17:55
@GrumpyCrouton PHP can absolutely be infinite running, if run via the CLI.
– ceejayoz
Nov 9 at 17:55
@ceejayoz Interesting, thank you.
– GrumpyCrouton
Nov 9 at 18:37
@ceejayoz Interesting, thank you.
– GrumpyCrouton
Nov 9 at 18:37
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
There's more than one answer to this.
Yes, you could have cron poll regularly for new queue items. You could have a daemon running indefinitely (likely monitored by something like supervisor) that continues to poll in a loop.
There's also SQS triggers, where a new SQS item can automatically initiate something. There are multiple options available: new queue items can make an SNS notification, which could trigger a HTTP POST to a URL. They can also trigger a Lambda function.
answered Nov 9 at 17:58
ceejayoz
140k34215299
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
|
show 1 more comment
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53230654%2fhow-would-i-run-a-queue-of-an-amazon-sqs-is-it-a-cron-job%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
There's more than one answer to this.
Yes, you could have cron poll regularly for new queue items. You could have a daemon running indefinitely (likely monitored by something like supervisor) that continues to poll in a loop.
There's also SQS triggers, where a new SQS item can automatically initiate something. There are multiple options available: new queue items can make an SNS notification, which could trigger a HTTP POST to a URL. They can also trigger a Lambda function.
answered Nov 9 at 17:58
ceejayoz
140k34215299
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
|
show 1 more comment
up vote
0
down vote
There's more than one answer to this.
Yes, you could have cron poll regularly for new queue items. You could have a daemon running indefinitely (likely monitored by something like supervisor) that continues to poll in a loop.
There's also SQS triggers, where a new SQS item can automatically initiate something. There are multiple options available: new queue items can make an SNS notification, which could trigger a HTTP POST to a URL. They can also trigger a Lambda function.
answered Nov 9 at 17:58
ceejayoz
140k34215299
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
|
show 1 more comment
up vote
0
down vote
up vote
0
down vote
There's more than one answer to this.
Yes, you could have cron poll regularly for new queue items. You could have a daemon running indefinitely (likely monitored by something like supervisor) that continues to poll in a loop.
There's also SQS triggers, where a new SQS item can automatically initiate something. There are multiple options available: new queue items can make an SNS notification, which could trigger a HTTP POST to a URL. They can also trigger a Lambda function.
answered Nov 9 at 17:58
ceejayoz
140k34215299
There's more than one answer to this.
Yes, you could have cron poll regularly for new queue items. You could have a daemon running indefinitely (likely monitored by something like supervisor) that continues to poll in a loop.
There's also SQS triggers, where a new SQS item can automatically initiate something. There are multiple options available: new queue items can make an SNS notification, which could trigger a HTTP POST to a URL. They can also trigger a Lambda function.
answered Nov 9 at 17:58
ceejayoz
140k34215299
answered Nov 9 at 17:58
ceejayoz
140k34215299
answered Nov 9 at 17:58
ceejayoz
140k34215299
answered Nov 9 at 17:58
ceejayoz
140k34215299
140k34215299
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
|
show 1 more comment
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
"new SQS item" .. just to be clear, you mean SQS queue triggers SNS topc every time a message is sent to the queue?
– Faisal
Nov 9 at 22:47
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
also if that's the case when will I ever need a process that keeps the queue running?
– Faisal
Nov 9 at 22:48
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@ceejayoz check that: The Lambda service can now poll an SQS queue and invoke a function, but SQS queues don't have any other types of triggers. You can do SNS > SQS but not the other way around.
– Michael - sqlbot
Nov 10 at 2:12
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
@Michael-sqlbot I'm sorry. But do you mean that SNS does subscribe to SQS? or not? does it trigger every-time a message is sent?
– Faisal
Nov 11 at 13:50
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
You cannot trigger SNS from SQS, @Faisal. You can do SNS → Lambda, SNS → SQS, and SQS → Lambda... but not SQS → SNS. The way AWS talks about how these service integrate is sometimes backwards to the way the integration is actually implemented. Technically, SQS is not actually capable of triggering anything -- but the Lambda service can be configured to set up managed listeners (on invisible servers they provide for free) to consume messages from an SQS queue and invoke a Lambda function with each message received... which behaves effectively as if SQS is triggering the Lambda function.
– Michael - sqlbot
Nov 11 at 15:51
|
show 1 more comment
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53230654%2fhow-would-i-run-a-queue-of-an-amazon-sqs-is-it-a-cron-job%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
I don't think that PHP could be "infinite running"
– GrumpyCrouton
Nov 9 at 17:34
Use a process manager like supervisord to keep one or more worker scripts running, and in your worker keep track of how many jobs you've processed and shut down after 1000 or so to make sure PHP's resource usage doesn't go nuts. Supervisord will make sure that there are always workers running.
– Sammitch
Nov 9 at 17:38
1
@GrumpyCrouton PHP can absolutely be infinite running, if run via the CLI.
– ceejayoz
Nov 9 at 17:55
@ceejayoz Interesting, thank you.
– GrumpyCrouton
Nov 9 at 18:37