How to run airflow script after previous airflow script is complete?
up vote
0
down vote
favorite
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
197
add a comment |
up vote
0
down vote
favorite
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
197
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
197
I know that for example if we have several sql files in the data folder we can set priorities like script1 >> script2 in the end.
However, can we set priorities of other airflow scripts which are actually in the dag folder?
python
google-cloud-platform airflow airflow-scheduler python
google-cloud-platform airflow airflow-scheduler asked Nov 8 at 11:29
pepperjohn
197
asked Nov 8 at 11:29
pepperjohn
197
asked Nov 8 at 11:29
pepperjohn
197
asked Nov 8 at 11:29
pepperjohn
197
asked Nov 8 at 11:29
pepperjohn
197
197
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
up vote
2
down vote
up vote
2
down vote
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
Those are not priorities, those are execution order. If you want a particular DAG to execute first then just set it at an earlier execution time than other DAGs.
script1 >> script2 means that script1 executes first and only then script2 executes.
You can try to artificially enforce priority by assigning a specific DAG to a named queue where it is the only DAG that executes in that named queue. Other than that there is no explicit way to have priority for entire DAGs.
For an Operator inside a DAG however you can use the priority_weight parameter in the BaseOperator.
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
answered Nov 9 at 4:27
Meghdeep Ray
2,50321738
2,50321738
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
Thanks Medhdeep that makes sense!
– pepperjohn
Nov 9 at 17:02
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53206852%2fhow-to-run-airflow-script-after-previous-airflow-script-is-complete%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
