Airflow Dag Patterns

by wshobson46891e7e60daNo licenseListed Oct 8, 2026Updated Oct 8, 2026

Build production Apache Airflow DAGs with best practices for operators, sensors, testing, and deployment. Use when creating data pipelines, orchestrating workflows, or scheduling batch jobs.

Instructions onlyDevOps & Cloud Software Development

AI-generated overview

Guides building production Apache Airflow DAGs with patterns for operators, sensors, testing, and deployment.

What it does: Provides production-oriented guidance for designing Apache Airflow DAGs, covering DAG design principles, task dependencies, operators, sensors, testing, and deployment strategies. It includes a quick-start example DAG and best-practice do's and don'ts, with further pattern documentation in a reference file. It produces instructions and code examples rather than running anything itself.
When to use it: Use when creating data pipeline orchestration, designing DAG structures and dependencies, implementing custom operators or sensors, testing DAGs locally, setting up Airflow in production, or debugging failed DAG runs.
Requirements: No scripts are shipped; it is instructions only. Following the examples assumes an Apache Airflow environment and Python, but nothing is required to read the guidance.

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

When to Use This Skill

Creating data pipeline orchestration with Airflow
Designing DAG structures and dependencies
Implementing custom operators and sensors
Testing Airflow DAGs locally
Setting up Airflow in production
Debugging failed DAG runs

Core Concepts

1. DAG Design Principles

2. Task Dependencies

python

# Lineartask1 >> task2 >> task3
# Fan-outtask1 >> [task2, task3, task4]
# Fan-in[task1, task2, task3] >> task4
# Complextask1 >> task2 >> task4task1 >> task3 >> task4

# Lineartask1 >> task2 >> task3
# Fan-outtask1 >> [task2, task3, task4]
# Fan-in[task1, task2, task3] >> task4
# Complextask1 >> task2 >> task4task1 >> task3 >> task4

Quick Start

python

# dags/example_dag.pyfrom datetime import datetime, timedeltafrom airflow import DAGfrom airflow.operators.python import PythonOperatorfrom airflow.operators.empty import EmptyOperator
default_args = {    'owner': 'data-team',    'depends_on_past': False,    'email_on_failure': True,    'email_on_retry': False,    'retries': 3,    'retry_delay': timedelta(minutes=5),    'retry_exponential_backoff': True,    'max_retry_delay': timedelta(hours=1),}
with DAG(    dag_id='example_etl',    default_args=default_args,    description='Example ETL pipeline',    schedule='0 6 * * *',  # Daily at 6 AM    start_date=datetime(2024, 1, 1),    catchup=False,    tags=['etl', 'example'],    max_active_runs=1,) as dag:
    start = EmptyOperator(task_id='start')
    def extract_data(**context):        execution_date = context['ds']        # Extract logic here        return {'records': 1000}
    extract = PythonOperator(        task_id='extract',        python_callable=extract_data,    )
    end = EmptyOperator(task_id='end')
    start >> extract >> end

# dags/example_dag.pyfrom datetime import datetime, timedeltafrom airflow import DAGfrom airflow.operators.python import PythonOperatorfrom airflow.operators.empty import EmptyOperator
default_args = {    'owner': 'data-team',    'depends_on_past': False,    'email_on_failure': True,    'email_on_retry': False,    'retries': 3,    'retry_delay': timedelta(minutes=5),    'retry_exponential_backoff': True,    'max_retry_delay': timedelta(hours=1),}
with DAG(    dag_id='example_etl',    default_args=default_args,    description='Example ETL pipeline',    schedule='0 6 * * *',  # Daily at 6 AM    start_date=datetime(2024, 1, 1),    catchup=False,    tags=['etl', 'example'],    max_active_runs=1,) as dag:
    start = EmptyOperator(task_id='start')
    def extract_data(**context):        execution_date = context['ds']        # Extract logic here        return {'records': 1000}
    extract = PythonOperator(        task_id='extract',        python_callable=extract_data,    )
    end = EmptyOperator(task_id='end')
    start >> extract >> end

Detailed patterns and worked examples

Detailed pattern documentation lives in references/details.md. Read that file when the navigation tier above is insufficient.

Best Practices

Do's

Use TaskFlow API - Cleaner code, automatic XCom
Set timeouts - Prevent zombie tasks
Use mode='reschedule' - For sensors, free up workers
Test DAGs - Unit tests and integration tests
Idempotent tasks - Safe to retry

Don'ts

Don't use depends_on_past=True - Creates bottlenecks
Don't hardcode dates - Use {{ ds }} macros
Don't use global state - Tasks should be stateless
Don't skip catchup blindly - Understand implications
Don't put heavy logic in DAG file - Import from modules

Source and attribution

Source:wshobson/agentsinplugins/data-engineering/skills/airflow-dag-patternsat commit46891e7

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Principle	Description
Idempotent	Running twice produces same result
Atomic	Tasks succeed or fail completely
Incremental	Process only new/changed data
Observable	Logs, metrics, alerts at every step

Airflow Dag Patterns

Apache Airflow DAG Patterns

When to Use This Skill

Core Concepts

1. DAG Design Principles

2. Task Dependencies

Quick Start

Detailed patterns and worked examples

Best Practices

Do's

Don'ts

Source and attribution

More from wshobson/agents

Uv Package Manager

Web Component Design

Python Type Safety

Visual Design Foundations

Responsive Design

Python Resource Management

More in DevOps & Cloud

M5 Onboard

Spanner Basics

Secops Cases

Managed Airflow Migrations

Managed Airflow Dag Troubleshooting

Iam Helper For Privileged Access Management