YAML Workflow

YAML Workflow

[PyPI version] [Python versions] [CI] [codecov] [License: MIT]

A lightweight workflow engine for CI/CD pipelines, data processing, and DevOps automation. Define reproducible, version-controlled workflows in YAML — run them locally, in CI, or on any machine with Python installed.

Why yaml-workflow?

Most workflow tools require servers, databases, and complex infrastructure. yaml-workflow takes a GitOps approach — workflows are plain YAML files, version-controlled alongside your code:

Choose yaml-workflow when you need:

• Simple task automation without infrastructure overhead
• Reproducible pipelines defined in version-controlled YAML
• Batch processing with parallel execution
• State persistence and workflow resume after failures
• A lightweight alternative to shell scripts with better error handling
• GitOps-friendly pipelines that live in your repo alongside the code
• A single tool that runs the same pipeline locally and in CI

Features

• YAML-driven workflow definition with Jinja2 templating
• Multiple task types: shell, Python, file, template, HTTP, batch
• Workflow composition via imports — reuse steps across workflows
• Plugin system via entry points — pip install yaml-workflow-myplugin
• Watch mode — --watch to re-run on file changes
• Dry-run mode to preview without executing
• Workflow visualization (ASCII branching DAG and Mermaid)
• Parallel execution with configurable worker pools
• State persistence and resume capability
• Retry mechanisms with configurable strategies
• Namespaced variables (args, env, steps, batch)
• Flow control with custom step sequences and conditions
• Extensible task system via @register_task decorator
• Parallel step execution via depends_on — run independent steps concurrently
• Secrets validation — fail fast if required environment variables are missing
• Structured output (--format json) for CI integration and scripting
• MCP server — expose workflows as AI agent tools (pip install yaml-workflow[mcp])
• Web dashboard — monitor runs and trigger workflows (pip install yaml-workflow[serve])
• GitHub Action — run workflows in CI with uses: orieg/[email protected]

Use Cases

• CI/CD pipelines — multi-step build, test, deploy workflows in YAML
• Data processing — batch ETL pipelines with retry and resume on failure
• DevOps automation — infrastructure tasks with secrets management and notifications
• AI/LLM pipelines — orchestrate API calls with auth, retry, and batch processing
• Local automation — replace shell scripts with reproducible, parameterized workflows

Quick Start

# Install (isolated CLI — recommended)pipx install yaml-workflow            # Core CLIpipx install 'yaml-workflow[all]'     # + web dashboard + MCP server
# Or with pippip install yaml-workflow
# Initialize example workflowsyaml-workflow init
# Run a workflow with parametersyaml-workflow run workflows/hello_world.yaml name=Alice

Example workflow (hello_world.yaml):

name: Hello Worlddescription: A simple greeting workflow
params:  name:    type: string    default: World
steps:  - name: create_greeting    task: template    inputs:      template: "Hello, {{ args.name }}!"      output_file: greeting.txt
  - name: show_greeting    task: shell    inputs:      command: cat greeting.txt

Visualize workflows

yaml-workflow visualize workflows/data_pipeline.yaml

  Workflow: Data Pipeline
  ┌─────────────────┐  │  detect_format  │  │   python_code   │  └─────────────────┘           │           ▼  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐  │  process_json  │  │  process_csv   │  │  process_xml   │  │ handle_unknown │  │     shell      │  │     shell      │  │     shell      │  │     shell      │  └────────────────┘  └────────────────┘  └────────────────┘  └────────────────┘           │           ▼  ┌─────────────────┐  │ generate_report │  │   python_code   │  └─────────────────┘

Adjacent conditional steps are automatically grouped as branches. Use --format mermaid to export for docs or GitHub rendering.

Dry-run mode

Preview what a workflow would do without executing anything:

yaml-workflow run workflows/hello_world.yaml name=Alice --dry-run

[DRY-RUN] Workflow: Hello World[DRY-RUN] Steps: 2 to execute
  [DRY-RUN] Step 'create_greeting' — task: template — WOULD EXECUTE    template: Hello, Alice!    output_file: greeting.txt  [DRY-RUN] Step 'show_greeting' — task: shell — WOULD EXECUTE    command: cat greeting.txt
[DRY-RUN] Complete. 2 step(s) would execute, 0 would be skipped.[DRY-RUN] No files were written. No tasks were executed.

Workflow composition

Reuse steps across workflows with imports:

# main.yamlimports:  - ./shared/logging_steps.yaml  - ./shared/common_params.yaml
steps:  - name: my_step    task: shell    inputs:      command: echo "runs after imported steps"

Imported steps are prepended. Imported params provide defaults that the main workflow can override. Supports transitive imports with circular detection.

Parallel Steps

Run independent steps concurrently with depends_on:

steps:  - name: fetch_api    task: http.request    inputs: {url: "https://api.example.com/data"}
  - name: fetch_db    task: python_code    inputs:      code: "result = query_database()"
  - name: merge    task: python_code    depends_on: [fetch_api, fetch_db]    inputs:      code: |        api_data = steps["fetch_api"]["result"]        db_data = steps["fetch_db"]["result"]        result = {"merged": True}

Watch mode

Automatically re-run on file changes during development:

yaml-workflow run workflows/hello_world.yaml name=Alice --watch

Monitors the workflow file and all imported files. Press Ctrl+C to stop.

GitHub Actions

Run workflows in CI with the yaml-workflow action:

- name: Run pipeline  uses: orieg/[email protected]  id: pipeline  with:    workflow: workflows/deploy.yaml    params: |      env=production      version=1.2.0    format: json
- name: Use results  run: echo '${{ steps.pipeline.outputs.result }}'

Docker & Kubernetes

Run anywhere without installing Python:

# Run a workflow in Dockerdocker run --rm -v $(pwd)/workflows:/app/workflows \  ghcr.io/orieg/yaml-workflow run /app/workflows/pipeline.yaml
# Start the web dashboarddocker run -p 8080:8080 -v $(pwd)/workflows:/app/workflows \  ghcr.io/orieg/yaml-workflow

Deploy on Kubernetes with the Helm chart:

helm install my-workflows ./helm/yaml-workflow \  --set-file workflows.files.pipeline\\.yaml=workflows/pipeline.yaml

Compatible with ArgoCD (GitOps) and Argo Workflows. See the Kubernetes guide.

More commands

# List available workflowsyaml-workflow list
# Validate a workflow (with JSON output for CI)yaml-workflow validate workflows/hello_world.yaml --format json
# Resume a failed workflowyaml-workflow run workflows/hello_world.yaml --resume
# Structured output for scriptingyaml-workflow run workflows/pipeline.yaml --format json --output results.json

Documentation

Full documentation is available at orieg.github.io/yaml-workflow.

• Getting Started - Installation and first workflow
• Task Types - Shell, Python, file, template, and batch tasks
• Workflow Structure - YAML configuration reference
• Templating - Jinja2 variable substitution
• State Management - Persistence and resume
• Task Development - Creating custom tasks
• API Reference - Full API documentation

Contributing

Contributions are welcome! See the Contributing Guide for development setup and guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.