Infrastructure Cicd Data Engineering

reason-machines/data-skills/skills/infrastructure-cicd-data-engineering

by reason-machines8e7bd5f22574No license4 starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 2 months ago

Deploy data infrastructure changes using CI/CD patterns with GitHub Actions, Terraform, and AWS OIDC authentication

Instructions onlyDevOps & Cloud
AI-generated overview

Sets up CI/CD pipelines that deploy AWS data infrastructure with Terraform and GitHub Actions using OIDC.

What it does
This skill provides instructions and configuration examples for building a CI/CD pipeline that provisions AWS data infrastructure with Terraform and GitHub Actions. It covers bootstrapping an S3 state backend, DynamoDB locking, and an IAM OIDC provider, then defining CI validation and plan/apply deployment workflows with manual production approval. It also includes example data resources such as S3 data lakes, Glue catalog databases, and Kinesis streams, plus drift detection, cost estimation, troubleshooting, and teardown guidance.
When to use it
Use it when you need to automate Terraform deployments of AWS data infrastructure through GitHub Actions. It fits teams wanting keyless OIDC authentication, pull-request validation, and gated production applies. It also suits adding drift detection or cost estimation to an existing Terraform pipeline.
Requirements
Requires an AWS account with permissions, Terraform v1.0+ installed locally, AWS CLI configured with credentials, and a GitHub repository. GitHub Actions secrets such as AWS_ROLE_ARN and a protected production environment are needed. The skill is instructions only and ships no scripts, though it references a tear-down.sh example.

Infrastructure CI/CD for Data Engineering

Skill by ara.so — Data Skills collection

This project demonstrates practical CI/CD patterns for deploying data infrastructure changes using GitHub Actions, Terraform, and AWS. It uses OpenID Connect (OIDC) for secure, keyless authentication between GitHub Actions and AWS, eliminating the need for long-lived AWS credentials.

What This Project Does

  • Bootstraps infrastructure: Creates S3 backend for Terraform state and OIDC provider for GitHub Actions
  • Automates deployments: Uses GitHub Actions workflows to plan and apply Terraform changes
  • Enforces reviews: Requires manual approval before production deployments
  • Validates code: Runs Terraform formatting and validation checks on PRs

Project Structure

.├── terraform/│   ├── bootstrap/          # Initial setup (S3 backend, OIDC)│   │   └── main.tf│   └── main/              # Main infrastructure definitions│       └── main.tf├── .github/│   └── workflows/│       ├── ci.yml         # Format and validation checks│       └── deploy.yml     # Deployment workflow└── tear-down.sh           # Cleanup script

Prerequisites

  1. AWS Account with appropriate permissions
  2. Terraform installed locally (v1.0+)
  3. GitHub Account and repository access
  4. AWS CLI configured with credentials
bash
# Verify Terraform installationterraform version
# Verify AWS credentialsaws sts get-caller-identity

Bootstrap Setup

Step 1: Create S3 Backend and OIDC Provider

The bootstrap process creates:

  • S3 bucket for Terraform state storage
  • DynamoDB table for state locking
  • IAM OIDC provider for GitHub Actions
  • IAM role that GitHub Actions will assume
bash
# Initialize and apply bootstrap configurationterraform -chdir=terraform/bootstrap initterraform -chdir=terraform/bootstrap apply
# Capture the outputsterraform -chdir=terraform/bootstrap output

Expected output:

github_actions_role_arn = "arn:aws:iam::123456789012:role/github-actions-role"state_bucket_name = "my-terraform-state-bucket"

Step 2: Configure GitHub Repository Secrets

Create a repository secret named AWS_ROLE_ARN:

  1. Navigate to: Settings → Secrets and variables → Actions → New repository secret
  2. Name: AWS_ROLE_ARN
  3. Value: The ARN output from bootstrap (without quotes)
bash
# Example ARN format (don't include quotes when pasting)arn:aws:iam::123456789012:role/github-actions-role

Step 3: Create GitHub Environment

Set up a production environment with manual approval:

  1. Navigate to: Settings → Environments → New environment
  2. Name: production
  3. Configure protection rules:
    • ✅ Required reviewers (minimum 1)
    • Add yourself or team members as reviewers

Bootstrap Terraform Configuration

terraform/bootstrap/main.tf (simplified example):

hcl
terraform {  required_version = ">= 1.0"  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = var.aws_region}
# S3 bucket for Terraform stateresource "aws_s3_bucket" "terraform_state" {  bucket = "${var.project_name}-terraform-state-${var.environment}"    tags = {    Name        = "Terraform State Bucket"    Environment = var.environment    ManagedBy   = "Terraform"  }}
resource "aws_s3_bucket_versioning" "terraform_state" {  bucket = aws_s3_bucket.terraform_state.id    versioning_configuration {    status = "Enabled"  }}
# DynamoDB table for state lockingresource "aws_dynamodb_table" "terraform_locks" {  name         = "${var.project_name}-terraform-locks"  billing_mode = "PAY_PER_REQUEST"  hash_key     = "LockID"    attribute {    name = "LockID"    type = "S"  }    tags = {    Name        = "Terraform State Lock Table"    Environment = var.environment    ManagedBy   = "Terraform"  }}
# OIDC provider for GitHub Actionsresource "aws_iam_openid_connect_provider" "github_actions" {  url = "https://token.actions.githubusercontent.com"    client_id_list = [    "sts.amazonaws.com"  ]    thumbprint_list = [    "6938fd4d98bab03faadb97b34396831e3780aea1"  ]}
# IAM role for GitHub Actionsresource "aws_iam_role" "github_actions" {  name = "github-actions-terraform-role"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Principal = {          Federated = aws_iam_openid_connect_provider.github_actions.arn        }        Action = "sts:AssumeRoleWithWebIdentity"        Condition = {          StringEquals = {            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"          }          StringLike = {            "token.actions.githubusercontent.com:sub" = "repo:${var.github_org}/${var.github_repo}:*"          }        }      }    ]  })}
# Attach policies to the roleresource "aws_iam_role_policy_attachment" "github_actions_admin" {  role       = aws_iam_role.github_actions.name  policy_arn = "arn:aws:iam::aws:policy/AdministratorAccess"}
# Outputsoutput "github_actions_role_arn" {  value       = aws_iam_role.github_actions.arn  description = "ARN of the IAM role for GitHub Actions"}
output "state_bucket_name" {  value       = aws_s3_bucket.terraform_state.bucket  description = "Name of the S3 bucket for Terraform state"}
output "state_lock_table_name" {  value       = aws_dynamodb_table.terraform_locks.name  description = "Name of the DynamoDB table for state locking"}

terraform/bootstrap/variables.tf:

hcl
variable "aws_region" {  description = "AWS region for resources"  type        = string  default     = "us-east-1"}
variable "project_name" {  description = "Project name for resource naming"  type        = string  default     = "data-infra"}
variable "environment" {  description = "Environment name"  type        = string  default     = "production"}
variable "github_org" {  description = "GitHub organization or username"  type        = string}
variable "github_repo" {  description = "GitHub repository name"  type        = string}

Main Infrastructure Configuration

terraform/main/main.tf (example data infrastructure):

hcl
terraform {  required_version = ">= 1.0"    backend "s3" {    bucket         = "data-infra-terraform-state-production"    key            = "main/terraform.tfstate"    region         = "us-east-1"    dynamodb_table = "data-infra-terraform-locks"    encrypt        = true  }    required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = var.aws_region    default_tags {    tags = {      Project     = var.project_name      Environment = var.environment      ManagedBy   = "Terraform"      DeployedBy  = "GitHub-Actions"    }  }}
# Example: S3 bucket for data lakeresource "aws_s3_bucket" "data_lake" {  bucket = "${var.project_name}-data-lake-${var.environment}"}
resource "aws_s3_bucket_versioning" "data_lake" {  bucket = aws_s3_bucket.data_lake.id    versioning_configuration {    status = "Enabled"  }}
resource "aws_s3_bucket_server_side_encryption_configuration" "data_lake" {  bucket = aws_s3_bucket.data_lake.id    rule {    apply_server_side_encryption_by_default {      sse_algorithm = "AES256"    }  }}
# Example: Glue database for data catalogresource "aws_glue_catalog_database" "analytics" {  name        = "${var.project_name}_analytics_${var.environment}"  description = "Analytics data catalog database"}
# Example: IAM role for Glue jobsresource "aws_iam_role" "glue_job" {  name = "${var.project_name}-glue-job-role-${var.environment}"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Principal = {          Service = "glue.amazonaws.com"        }        Action = "sts:AssumeRole"      }    ]  })}
resource "aws_iam_role_policy_attachment" "glue_service" {  role       = aws_iam_role.glue_job.name  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole"}
# Outputsoutput "data_lake_bucket" {  value       = aws_s3_bucket.data_lake.bucket  description = "Name of the data lake S3 bucket"}
output "glue_database" {  value       = aws_glue_catalog_database.analytics.name  description = "Name of the Glue catalog database"}

GitHub Actions Workflows

CI Workflow: Format and Validation

.github/workflows/ci.yml:

yaml
name: Terraform CI
on:  pull_request:    branches:      - main    paths:      - 'terraform/**'      - '.github/workflows/ci.yml'
permissions:  contents: read  pull-requests: write
jobs:  terraform-checks:    name: Terraform Checks    runs-on: ubuntu-latest        steps:      - name: Checkout code        uses: actions/checkout@v4            - name: Setup Terraform        uses: hashicorp/setup-terraform@v3        with:          terraform_version: 1.5.0            - name: Terraform Format Check        id: fmt        run: terraform fmt -check -recursive terraform/        continue-on-error: true            - name: Terraform Init (Main)        run: terraform -chdir=terraform/main init -backend=false            - name: Terraform Validate (Main)        run: terraform -chdir=terraform/main validate            - name: Comment PR        if: steps.fmt.outcome == 'failure'        uses: actions/github-script@v7        with:          script: |            github.rest.issues.createComment({              issue_number: context.issue.number,              owner: context.repo.owner,              repo: context.repo.repo,              body: '❌ Terraform formatting check failed. Run `terraform fmt -recursive terraform/` to fix.'            })            - name: Fail if format check failed        if: steps.fmt.outcome == 'failure'        run: exit 1

Deploy Workflow: Plan and Apply

.github/workflows/deploy.yml:

yaml
name: Deploy Infrastructure
on:  push:    branches:      - main    paths:      - 'terraform/main/**'  workflow_dispatch:
permissions:  id-token: write  contents: read  pull-requests: write
jobs:  terraform-plan:    name: Terraform Plan    runs-on: ubuntu-latest        steps:      - name: Checkout code        uses: actions/checkout@v4            - name: Configure AWS credentials        uses: aws-actions/configure-aws-credentials@v4        with:          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}          aws-region: us-east-1            - name: Setup Terraform        uses: hashicorp/setup-terraform@v3        with:          terraform_version: 1.5.0            - name: Terraform Init        run: terraform -chdir=terraform/main init            - name: Terraform Plan        id: plan        run: |          terraform -chdir=terraform/main plan -no-color -out=tfplan          terraform -chdir=terraform/main show -no-color tfplan > plan.txt            - name: Upload plan        uses: actions/upload-artifact@v4        with:          name: terraform-plan          path: |            terraform/main/tfplan            plan.txt          retention-days: 5    terraform-apply:    name: Terraform Apply    needs: terraform-plan    runs-on: ubuntu-latest    environment: production        steps:      - name: Checkout code        uses: actions/checkout@v4            - name: Configure AWS credentials        uses: aws-actions/configure-aws-credentials@v4        with:          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}          aws-region: us-east-1            - name: Setup Terraform        uses: hashicorp/setup-terraform@v3        with:          terraform_version: 1.5.0            - name: Terraform Init        run: terraform -chdir=terraform/main init            - name: Download plan        uses: actions/download-artifact@v4        with:          name: terraform-plan          path: terraform/main/            - name: Terraform Apply        run: terraform -chdir=terraform/main apply -auto-approve tfplan

Common Workflows

Adding New Infrastructure

  1. Create/modify Terraform files in terraform/main/:
hcl
# terraform/main/kinesis.tfresource "aws_kinesis_stream" "events" {  name             = "${var.project_name}-events-${var.environment}"  shard_count      = 1  retention_period = 24    shard_level_metrics = [    "IncomingBytes",    "IncomingRecords",    "OutgoingBytes",    "OutgoingRecords",  ]}
output "kinesis_stream_name" {  value       = aws_kinesis_stream.events.name  description = "Name of the Kinesis stream"}
  1. Format Terraform files:
bash
terraform fmt -recursive terraform/
  1. Validate locally (optional but recommended):
bash
terraform -chdir=terraform/main init -backend=falseterraform -chdir=terraform/main validate

  1. Create a pull request:

    • CI workflow runs format check and validation
    • Review the checks before merging

  2. Merge to main:

    • Deploy workflow runs terraform plan
    • Manual approval required in GitHub UI
    • After approval, terraform apply executes

Checking Deployment Status

bash
# View workflow runsgh run list --workflow=deploy.yml
# View specific run logsgh run view <run-id> --log
# Check specific jobgh run view <run-id> --job=<job-id>

Testing Changes Locally

bash
# Initialize with backendterraform -chdir=terraform/main init
# Plan changesterraform -chdir=terraform/main plan
# Apply (be careful in production!)terraform -chdir=terraform/main apply

Environment Variables and Configuration

Required GitHub Secrets

Secret NameDescriptionExample
AWS_ROLE_ARNIAM role ARN for GitHub Actionsarn:aws:iam::123456789012:role/github-actions-role

Terraform Variables

Create terraform/main/terraform.tfvars:

hcl
aws_region   = "us-east-1"project_name = "my-data-platform"environment  = "production"
# Additional configurationenable_monitoring = truedata_retention_days = 90

Using Environment-Specific Configurations

terraform/main/environments/dev.tfvars:

hcl
environment  = "dev"project_name = "my-data-platform"aws_region   = "us-east-1"
# Dev-specific settingsenable_monitoring   = falsedata_retention_days = 7

terraform/main/environments/prod.tfvars:

hcl
environment  = "production"project_name = "my-data-platform"aws_region   = "us-east-1"
enable_monitoring   = truedata_retention_days = 90

Modify workflow to use environment-specific variables:

yaml
- name: Terraform Plan  run: |    terraform -chdir=terraform/main plan \      -var-file=environments/${{ github.event.inputs.environment }}.tfvars \      -out=tfplan

Advanced Patterns

Matrix Deployments for Multiple Environments

yaml
jobs:  terraform-plan:    strategy:      matrix:        environment: [dev, staging, production]    name: Plan - ${{ matrix.environment }}    runs-on: ubuntu-latest    environment: ${{ matrix.environment }}        steps:      - uses: actions/checkout@v4            - uses: aws-actions/configure-aws-credentials@v4        with:          role-to-assume: ${{ secrets[format('AWS_ROLE_ARN_{0}', matrix.environment)] }}          aws-region: us-east-1            - uses: hashicorp/setup-terraform@v3            - name: Terraform Plan        run: |          terraform -chdir=terraform/main plan \            -var-file=environments/${{ matrix.environment }}.tfvars \            -out=tfplan-${{ matrix.environment }}

Drift Detection Scheduled Job

.github/workflows/drift-detection.yml:

yaml
name: Terraform Drift Detection
on:  schedule:    - cron: '0 6 * * *'  # Daily at 6 AM UTC  workflow_dispatch:
permissions:  id-token: write  contents: read  issues: write
jobs:  detect-drift:    name: Detect Infrastructure Drift    runs-on: ubuntu-latest        steps:      - uses: actions/checkout@v4            - uses: aws-actions/configure-aws-credentials@v4        with:          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}          aws-region: us-east-1            - uses: hashicorp/setup-terraform@v3            - name: Terraform Init        run: terraform -chdir=terraform/main init            - name: Terraform Plan        id: plan        run: |          terraform -chdir=terraform/main plan -detailed-exitcode -no-color > plan.txt        continue-on-error: true            - name: Create Issue on Drift        if: steps.plan.outputs.exitcode == 2        uses: actions/github-script@v7        with:          script: |            const fs = require('fs');            const plan = fs.readFileSync('plan.txt', 'utf8');                        await github.rest.issues.create({              owner: context.repo.owner,              repo: context.repo.repo,              title: '⚠️ Infrastructure Drift Detected',              body: `Drift detected in Terraform state.\n\n\`\`\`\n${plan}\n\`\`\``,              labels: ['drift', 'infrastructure']            });

Cost Estimation with Infracost

Add to .github/workflows/ci.yml:

yaml
- name: Setup Infracost  uses: infracost/actions/setup@v2  with:    api-key: ${{ secrets.INFRACOST_API_KEY }}
- name: Generate cost estimate  run: |    infracost breakdown \      --path=terraform/main \      --format=json \      --out-file=/tmp/infracost.json
- name: Post cost comment  run: |    infracost comment github \      --path=/tmp/infracost.json \      --repo=$GITHUB_REPOSITORY \      --github-token=${{ secrets.GITHUB_TOKEN }} \      --pull-request=${{ github.event.pull_request.number }}

Troubleshooting

"Error: configuring Terraform AWS Provider: failed to get shared config profile"

Solution: Ensure AWS credentials are properly configured in GitHub Actions:

yaml
- uses: aws-actions/configure-aws-credentials@v4  with:    role-to-assume: ${{ secrets.AWS_ROLE_ARN }}    aws-region: us-east-1

"Error: Error acquiring the state lock"

Cause: Another Terraform operation is running or a previous operation failed to release the lock.

Solution:

bash
# Force unlock (use with caution)terraform -chdir=terraform/main force-unlock <LOCK_ID>

"Error: InvalidClientTokenId: The security token included in the request is invalid"

Cause: OIDC provider not configured correctly or role ARN is incorrect.

Solution:

  1. Verify the AWS_ROLE_ARN secret matches bootstrap output
  2. Check OIDC provider trust policy includes your repository
  3. Ensure GitHub Actions has id-token: write permission

Format Check Failing

Error: Terraform files not properly formatted.

Solution:

bash
# Fix formatting locallyterraform fmt -recursive terraform/
# Check what would changeterraform fmt -check -recursive terraform/
# Commit and pushgit add terraform/git commit -m "fix: format terraform files"git push

State Backend Not Found

Error: "Error: Failed to get existing workspaces: S3 bucket does not exist"

Cause: Backend configuration references a bucket that doesn't exist.

Solution:

  1. Verify bootstrap was applied: terraform -chdir=terraform/bootstrap output
  2. Update backend configuration in terraform/main/main.tf with correct bucket name
  3. Re-run terraform init

Manual Approval Not Showing

Cause: Production environment not configured or reviewers not set.

Solution:

  1. Go to Settings → Environments → production
  2. Enable "Required reviewers"
  3. Add at least one reviewer
  4. Re-run the workflow

Cleanup

Destroy all resources:

bash
# Run the teardown script./tear-down.sh
# Or manuallyterraform -chdir=terraform/main destroyterraform -chdir=terraform/bootstrap destroy

tear-down.sh example:

bash
#!/bin/bashset -e
echo "Destroying main infrastructure..."terraform -chdir=terraform/main destroy -auto-approve
echo "Destroying bootstrap resources..."terraform -chdir=terraform/bootstrap destroy -auto-approve
echo "Cleanup complete!"

Best Practices

  1. Always run terraform fmt before committing
  2. Use meaningful commit messages that describe infrastructure changes
  3. Review plans carefully before approving deployments
  4. Enable branch protection on main branch
  5. Use separate AWS accounts for dev/staging/production
  6. Monitor state file changes for unauthorized modifications
  7. Implement drift detection to catch manual changes
  8. Version your Terraform providers to ensure consistency
  9. Use workspaces or separate backends for different environments
  10. Document custom modules and complex configurations

Source and attribution

Source:reason-machines/data-skillsinskills/infrastructure-cicd-data-engineeringat commit8e7bd5f

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

More from reason-machines/data-skills

Youtube Channel Analytics Dashboard

reason-machines

Describes a Windows dashboard tool for tracking YouTube channel metrics such as views, subscribers, revenue and demographics.

Data & Analytics4updated 2 months ago

Terraform Iac Data Engineering

reason-machines

Provides Terraform infrastructure-as-code patterns for provisioning AWS data engineering resources such as S3, EC2 and IAM.

DevOps & Cloud4updated 2 months ago

Terraform Data Engineering Infrastructure

reason-machines

Terraform patterns and commands for provisioning AWS data engineering infrastructure such as S3, EC2 and IAM.

DevOps & Cloud4updated 2 months ago

Retail Etl Pipeline Medallion

reason-machines

Guides building a retail ETL data warehouse with Bronze, Silver and Gold layers using SQL Server and optional PySpark.

Data & Analytics4updated 2 months ago

Power Bi Salespulse 360 Dashboard

reason-machines

Guides building the SalesPulse 360 Power BI retail dashboard for sales, profit, forecasting and alerts.

Data & Analytics4updated 2 months ago

Logifleet Pulse Supply Chain Analytics

reason-machines

Builds a logistics data warehouse on MS SQL Server with Power BI dashboards for warehouse and fleet analytics.

Data & Analytics4updated 2 months ago

More in DevOps & Cloud

Playwright Devops

microsoft

Featured

DevOps workflows for Playwright: analyze GitHub Actions failures for the last commit on main and fetch failed job logs.

Includes scripts
DevOps & Cloud97Kupdated today

M5 Onboard

anthropics

Featured

Provisions M5Stack ESP32 boards by detecting them on USB, flashing UIFlow 2.0 firmware, and installing a MicroPython app bundle.

DevOps & Cloud37Kupdated today

Yeet

openai

Featured

Stages, commits, pushes, and opens or updates a GitHub pull request in one flow using the GitHub CLI.

DevOps & Cloud27Kupdated 4 weeks ago

Runbook

anthropics

Featured

Creates or updates step-by-step operational runbooks for recurring tasks, including troubleshooting, rollback and escalation.

DevOps & Cloud27Kupdated today

Incident Response

anthropics

Featured

Guides an incident response workflow: severity triage, status updates, mitigation tracking, and blameless postmortems.

DevOps & Cloud27Kupdated today

Deploy Checklist

anthropics

Featured

Generates a pre-deployment readiness checklist covering pre-deploy, deploy, post-deploy and rollback triggers.

DevOps & Cloud27Kupdated today