Wsrtjtyk

import airflow

from airflow import DAG

from airflow.operators.python_operator import PythonOperator

from datetime import datetime, timedelta

from workflow.task import some_task



default_args = {

    'owner': 'airflow',

    'depends_on_past': False,

    'email': ['jimin.park1@aig.com'],

    'email_on_failure': False,

    'email_on_retry': False,

    'retries': 0,

    'retry_delay': timedelta(minutes=1),

    'start_date': airflow.utils.dates.days_ago(0)

    # 'queue': 'bash_queue',

    # 'pool': 'backfill',

    # 'priority_weight': 10,

    # 'end_date': datetime(2016, 1, 1),

}



dag = DAG('JiminTest', default_args=default_args, schedule_interval='*/1 * * * *', catchup=False)



t1 = PythonOperator(

    task_id='Task1',

    provide_context=True,

    python_callable=some_task,

    dag=dag

)

The actual some_task itself simply appends timestamp to some file. As you can see in the dag config file, the task itself is configured to run every 1 min.

def some_task(ds, **kwargs):

    current_time = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    with open("test.txt", "a") as myfile:

        myfile.write(current_time + 'n')

I simply tail -f the output file and started up the webserver without the scheduler running. This function was being called and things were being appended to the file when webserver starts up. When I start up the scheduler, on each execution loop, the file gets appended.

What I want is for the function to be executed on every minute as intended, not every execution loop.

The scheduler will run each DAG file every scheduler loop, including all import statements.

Is there anything running code in the file from where you are importing the function?

Try to check the scheduler_heartbeat_sec config parameter in your config file. For your case it should be smaller than 60 seconds.

If you want the scheduler not to cahtchup previous runs set catchup_by_defaultto False (I am not sure if this relevant to your question though).

Please indicate which Apache Airflow version are you using