To kill spark application
I wanted to kill spark application id if its running for more than specific hours for eg. one hour. Is there a way possible to achieve in unix script running in particular user id. User id - application_user
yarn application -kill application_ID
apache-spark unix
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
add a comment |
I wanted to kill spark application id if its running for more than specific hours for eg. one hour. Is there a way possible to achieve in unix script running in particular user id. User id - application_user
yarn application -kill application_ID
apache-spark unix
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
add a comment |
I wanted to kill spark application id if its running for more than specific hours for eg. one hour. Is there a way possible to achieve in unix script running in particular user id. User id - application_user
yarn application -kill application_ID
apache-spark unix
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
I wanted to kill spark application id if its running for more than specific hours for eg. one hour. Is there a way possible to achieve in unix script running in particular user id. User id - application_user
yarn application -kill application_ID
apache-spark unix
apache-spark unix
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
asked Nov 13 '18 at 4:07
user2672739user2672739
5428
5428
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
I can propose the next solution:
You can parse the time when the application was started from the applicationId.
(Use curl and Spark rest API to create a command to get application list and to take the correct applicationId: https://spark.apache.org/docs/latest/monitoring.html)
After this step, you will have a date in format YYYYMMDDHHMMSS (For example 20181112020100)Now you need to get the current time using the command:
date +%Y%m%d%H%M%SIn a cycle, get the current time and compare it with the time which you get from the first step which is application started time and compare it with the current time. When the difference between application started time and the current time will be, for example, two hours you will exit from a cycle and use kill command:
yarn application -kill application_ID
Here is one of the realizations of proposed steps: https://community.hortonworks.com/articles/86037/script-to-kill-yarn-application-if-it-is-running-m.html
Hope, this will be useful.
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273649%2fto-kill-spark-application%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I can propose the next solution:
You can parse the time when the application was started from the applicationId.
(Use curl and Spark rest API to create a command to get application list and to take the correct applicationId: https://spark.apache.org/docs/latest/monitoring.html)
After this step, you will have a date in format YYYYMMDDHHMMSS (For example 20181112020100)Now you need to get the current time using the command:
date +%Y%m%d%H%M%SIn a cycle, get the current time and compare it with the time which you get from the first step which is application started time and compare it with the current time. When the difference between application started time and the current time will be, for example, two hours you will exit from a cycle and use kill command:
yarn application -kill application_ID
Here is one of the realizations of proposed steps: https://community.hortonworks.com/articles/86037/script-to-kill-yarn-application-if-it-is-running-m.html
Hope, this will be useful.
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
add a comment |
I can propose the next solution:
You can parse the time when the application was started from the applicationId.
(Use curl and Spark rest API to create a command to get application list and to take the correct applicationId: https://spark.apache.org/docs/latest/monitoring.html)
After this step, you will have a date in format YYYYMMDDHHMMSS (For example 20181112020100)Now you need to get the current time using the command:
date +%Y%m%d%H%M%SIn a cycle, get the current time and compare it with the time which you get from the first step which is application started time and compare it with the current time. When the difference between application started time and the current time will be, for example, two hours you will exit from a cycle and use kill command:
yarn application -kill application_ID
Here is one of the realizations of proposed steps: https://community.hortonworks.com/articles/86037/script-to-kill-yarn-application-if-it-is-running-m.html
Hope, this will be useful.
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
add a comment |
I can propose the next solution:
You can parse the time when the application was started from the applicationId.
(Use curl and Spark rest API to create a command to get application list and to take the correct applicationId: https://spark.apache.org/docs/latest/monitoring.html)
After this step, you will have a date in format YYYYMMDDHHMMSS (For example 20181112020100)Now you need to get the current time using the command:
date +%Y%m%d%H%M%SIn a cycle, get the current time and compare it with the time which you get from the first step which is application started time and compare it with the current time. When the difference between application started time and the current time will be, for example, two hours you will exit from a cycle and use kill command:
yarn application -kill application_ID
Here is one of the realizations of proposed steps: https://community.hortonworks.com/articles/86037/script-to-kill-yarn-application-if-it-is-running-m.html
Hope, this will be useful.
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
I can propose the next solution:
You can parse the time when the application was started from the applicationId.
(Use curl and Spark rest API to create a command to get application list and to take the correct applicationId: https://spark.apache.org/docs/latest/monitoring.html)
After this step, you will have a date in format YYYYMMDDHHMMSS (For example 20181112020100)Now you need to get the current time using the command:
date +%Y%m%d%H%M%SIn a cycle, get the current time and compare it with the time which you get from the first step which is application started time and compare it with the current time. When the difference between application started time and the current time will be, for example, two hours you will exit from a cycle and use kill command:
yarn application -kill application_ID
Here is one of the realizations of proposed steps: https://community.hortonworks.com/articles/86037/script-to-kill-yarn-application-if-it-is-running-m.html
Hope, this will be useful.
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
answered Nov 13 '18 at 8:55
Yehor KrivokonYehor Krivokon
613214
613214
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53273649%2fto-kill-spark-application%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
