Terraform Iac Data Engineering

reason-machines/data-skills/skills/terraform-iac-data-engineering

作者 reason-machines8e7bd5f22574无许可证4 个星标收录于 2026年10月8日更新于 2026年10月8日仓库2个月前更新

Infrastructure-as-Code with Terraform for data engineering on AWS (S3, EC2, IAM)

仅含说明DevOps & Cloud
AI 生成的概览

提供用于在 AWS 上预置 S3、EC2 和 IAM 等数据工程资源的 Terraform 基础设施即代码模式。

功能
该技能提供使用 Terraform 为数据工程工作负载预置 AWS 基础设施的参考指导和配置示例。内容涵盖 S3 存储桶、EC2 实例、IAM 角色与策略、VPC 与安全组、远程状态后端、生命周期策略以及多环境模块结构。它还记录了 init、plan、apply、状态查看、导入、销毁和故障排查等 Terraform 命令。
适用场景
适用于使用 Terraform 搭建或管理数据管道的 AWS 基础设施，包括 S3 存储、EC2 处理实例和 IAM 访问权限。也适合处理状态管理、远程后端和多环境基础设施模式。
运行要求
需要 Terraform CLI 和 AWS CLI，并配置具有创建 S3、EC2、IAM、VPC 和 DynamoDB 资源权限的 AWS 凭证。需要访问 AWS 和 Terraform 提供程序注册表的网络连接。该技能仅为说明文档，不附带脚本。

Terraform IaC for Data Engineering

Skill by ara.so — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) patterns using Terraform specifically for data engineering workloads on AWS. It demonstrates how to provision and manage AWS resources (S3, EC2, IAM) needed for data pipelines and processing.

What This Project Does

  • Provisions AWS S3 buckets for data storage
  • Creates EC2 instances for data processing workloads
  • Manages IAM users, roles, and policies
  • Demonstrates Terraform state management
  • Provides reusable IaC patterns for data engineering infrastructure

Installation

Prerequisites

  1. Terraform CLI

    bash
    # macOSbrew install terraform
# Linuxwget https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_linux_amd64.zipunzip terraform_1.5.0_linux_amd64.zipsudo mv terraform /usr/local/bin/

  2. AWS CLI

    bash
    # macOSbrew install awscli
# Linuxcurl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"unzip awscliv2.zipsudo ./aws/install

  3. Configure AWS CLI

    bash
    aws configure# Enter your AWS Access Key ID# Enter your AWS Secret Access Key# Default region: us-east-1# Default output format: json

Project Setup

bash
git clone https://github.com/josephmachado/iac-for-data-engineering-terraform-.gitcd iac-for-data-engineering-terraform-

Key Terraform Commands

Initialize Terraform

bash
# Initialize terraform (downloads providers, sets up backend)terraform -chdir=terraform init
# Validate configuration filesterraform -chdir=terraform validate
# Format configuration filesterraform -chdir=terraform fmt

Plan and Apply Infrastructure

bash
# Preview changes before applyingterraform -chdir=terraform plan
# Apply infrastructure changesterraform -chdir=terraform apply
# Auto-approve without confirmation (use with caution)terraform -chdir=terraform apply -auto-approve

Inspect Infrastructure

bash
# List all resources in stateterraform -chdir=terraform state list
# Show details of a specific resourceterraform -chdir=terraform state show aws_s3_bucket.data_bucket
# Output specific valuesterraform -chdir=terraform output
# Show current state in JSONterraform -chdir=terraform show -json

Destroy Infrastructure

bash
# Destroy all managed infrastructureterraform -chdir=terraform destroy
# Destroy specific resourceterraform -chdir=terraform destroy -target=aws_instance.data_processor

Configuration Structure

Basic Terraform Configuration for Data Engineering

main.tf - Core infrastructure definition:

hcl
terraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = var.aws_region}
# S3 bucket for data storageresource "aws_s3_bucket" "data_lake" {  bucket = "my-unique-data-lake-bucket-${var.environment}"    tags = {    Name        = "Data Lake Bucket"    Environment = var.environment    Project     = "DataEngineering"  }}
# Enable versioning for data protectionresource "aws_s3_bucket_versioning" "data_lake_versioning" {  bucket = aws_s3_bucket.data_lake.id    versioning_configuration {    status = "Enabled"  }}
# Block public accessresource "aws_s3_bucket_public_access_block" "data_lake_public_access" {  bucket = aws_s3_bucket.data_lake.id    block_public_acls       = true  block_public_policy     = true  ignore_public_acls      = true  restrict_public_buckets = true}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = var.ec2_ami  instance_type = var.ec2_instance_type    tags = {    Name        = "DataProcessor"    Environment = var.environment  }    user_data = <<-EOF              #!/bin/bash              sudo yum update -y              sudo yum install -y python3 python3-pip              pip3 install pandas boto3              EOF}
# IAM role for EC2 to access S3resource "aws_iam_role" "ec2_s3_access_role" {  name = "ec2-s3-access-role"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })}
# IAM policy for S3 accessresource "aws_iam_role_policy" "ec2_s3_policy" {  name = "ec2-s3-policy"  role = aws_iam_role.ec2_s3_access_role.id    policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Action = [          "s3:GetObject",          "s3:PutObject",          "s3:ListBucket"        ]        Resource = [          aws_s3_bucket.data_lake.arn,          "${aws_s3_bucket.data_lake.arn}/*"        ]      }    ]  })}
# Attach IAM role to EC2 instanceresource "aws_iam_instance_profile" "ec2_profile" {  name = "ec2-s3-profile"  role = aws_iam_role.ec2_s3_access_role.name}

variables.tf - Input variables:

hcl
variable "aws_region" {  description = "AWS region for resources"  type        = string  default     = "us-east-1"}
variable "environment" {  description = "Environment name (dev, staging, prod)"  type        = string  default     = "dev"}
variable "ec2_ami" {  description = "AMI ID for EC2 instance"  type        = string  default     = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2}
variable "ec2_instance_type" {  description = "EC2 instance type"  type        = string  default     = "t2.micro"}
variable "bucket_prefix" {  description = "Prefix for S3 bucket names"  type        = string  default     = "data-eng"}

outputs.tf - Output values:

hcl
output "s3_bucket_name" {  description = "Name of the S3 data lake bucket"  value       = aws_s3_bucket.data_lake.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.data_lake.arn}
output "ec2_instance_id" {  description = "ID of the EC2 data processor"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of EC2 instance"  value       = aws_instance.data_processor.public_ip}

terraform.tfvars - Variable values (gitignore this file):

hcl
aws_region         = "us-west-2"environment        = "production"ec2_instance_type  = "t3.medium"bucket_prefix      = "my-company-data"

Common Data Engineering Patterns

Multi-Environment Setup

environments/dev/main.tf:

hcl
module "data_infrastructure" {  source = "../../modules/data-infra"    environment       = "dev"  instance_type     = "t2.micro"  enable_monitoring = false}

environments/prod/main.tf:

hcl
module "data_infrastructure" {  source = "../../modules/data-infra"    environment       = "prod"  instance_type     = "t3.xlarge"  enable_monitoring = true  backup_enabled    = true}

S3 Bucket with Lifecycle Policies

hcl
resource "aws_s3_bucket" "data_archive" {  bucket = "data-archive-${var.environment}"}
resource "aws_s3_bucket_lifecycle_configuration" "data_archive_lifecycle" {  bucket = aws_s3_bucket.data_archive.id    rule {    id     = "archive-old-data"    status = "Enabled"        transition {      days          = 30      storage_class = "STANDARD_IA"    }        transition {      days          = 90      storage_class = "GLACIER"    }        expiration {      days = 365    }  }    rule {    id     = "delete-incomplete-uploads"    status = "Enabled"        abort_incomplete_multipart_upload {      days_after_initiation = 7    }  }}

VPC Setup for Data Processing

hcl
resource "aws_vpc" "data_vpc" {  cidr_block           = "10.0.0.0/16"  enable_dns_hostnames = true  enable_dns_support   = true    tags = {    Name = "data-engineering-vpc"  }}
resource "aws_subnet" "private_subnet" {  vpc_id            = aws_vpc.data_vpc.id  cidr_block        = "10.0.1.0/24"  availability_zone = "${var.aws_region}a"    tags = {    Name = "private-data-subnet"  }}
resource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing instances"  vpc_id      = aws_vpc.data_vpc.id    egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }    ingress {    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["10.0.0.0/16"]  }}

Remote State Configuration

backend.tf:

hcl
terraform {  backend "s3" {    bucket         = "terraform-state-bucket-unique-name"    key            = "data-engineering/terraform.tfstate"    region         = "us-east-1"    encrypt        = true    dynamodb_table = "terraform-state-lock"  }}

Create state backend resources:

hcl
resource "aws_s3_bucket" "terraform_state" {  bucket = "terraform-state-bucket-unique-name"}
resource "aws_s3_bucket_versioning" "terraform_state_versioning" {  bucket = aws_s3_bucket.terraform_state.id    versioning_configuration {    status = "Enabled"  }}
resource "aws_dynamodb_table" "terraform_locks" {  name         = "terraform-state-lock"  billing_mode = "PAY_PER_REQUEST"  hash_key     = "LockID"    attribute {    name = "LockID"    type = "S"  }}

Verification and Testing

Verify S3 Bucket Creation

bash
# List all S3 bucketsaws s3 ls
# Check specific bucketaws s3 ls s3://my-unique-data-lake-bucket-dev/
# Upload test fileecho "test data" > test.txtaws s3 cp test.txt s3://my-unique-data-lake-bucket-dev/

Verify EC2 Instances

bash
# List running instancesaws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress}' \  --output table
# Get specific instance detailsaws ec2 describe-instances \  --instance-ids $(terraform -chdir=terraform output -raw ec2_instance_id)

Verify IAM Roles

bash
# List IAM rolesaws iam list-roles --query 'Roles[?contains(RoleName, `ec2-s3-access`)].RoleName'
# Get role policyaws iam get-role-policy \  --role-name ec2-s3-access-role \  --policy-name ec2-s3-policy

State Management

Inspect State

bash
# View state file (formatted)cat terraform/terraform.tfstate | jq -r '.resources[] | [.type, .name] | join(",")'
# List resources in stateterraform -chdir=terraform state list
# Show resource detailsterraform -chdir=terraform state show aws_s3_bucket.data_lake

Import Existing Resources

bash
# Import existing S3 bucketterraform -chdir=terraform import aws_s3_bucket.data_lake my-existing-bucket
# Import existing EC2 instanceterraform -chdir=terraform import aws_instance.data_processor i-1234567890abcdef0

Move Resources in State

bash
# Rename resource in stateterraform -chdir=terraform state mv aws_s3_bucket.old_name aws_s3_bucket.new_name

Troubleshooting

Common Issues

Issue: Bucket name already exists

hcl
# Solution: Use unique bucket name with random suffixresource "random_id" "bucket_suffix" {  byte_length = 4}
resource "aws_s3_bucket" "data_lake" {  bucket = "data-lake-${var.environment}-${random_id.bucket_suffix.hex}"}

Issue: AWS credentials not found

bash
# Check AWS configurationaws configure list
# Use environment variablesexport AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"export AWS_DEFAULT_REGION="us-east-1"

Issue: State file locked

bash
# Force unlock (use with caution)terraform -chdir=terraform force-unlock <LOCK_ID>

Issue: Resource already exists

bash
# Import existing resourceterraform -chdir=terraform import <resource_type>.<resource_name> <resource_id>
# Or remove from stateterraform -chdir=terraform state rm <resource_type>.<resource_name>

Issue: Terraform version mismatch

hcl
# Specify required version in terraform blockterraform {  required_version = ">= 1.5.0"    required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}

Debugging

bash
# Enable debug loggingexport TF_LOG=DEBUGterraform -chdir=terraform apply
# Log to fileexport TF_LOG_PATH=terraform-debug.logterraform -chdir=terraform apply
# Disable loggingunset TF_LOGunset TF_LOG_PATH

Validate and Format

bash
# Validate configurationterraform -chdir=terraform validate
# Format all filesterraform -chdir=terraform fmt -recursive
# Check formatting without making changesterraform -chdir=terraform fmt -check

Best Practices

  1. Always use variables for environment-specific values
  2. Enable S3 versioning for state files and data buckets
  3. Use remote state for team collaboration
  4. Tag all resources with environment, project, and owner
  5. Implement lifecycle policies for cost optimization
  6. Use modules for reusable infrastructure patterns
  7. Store secrets in AWS Secrets Manager, reference via data sources
  8. Run terraform plan before apply
  9. Use workspaces for multiple environments
  10. Document your infrastructure with comments and README files

来源与署名

来源：reason-machines/data-skills位于skills/terraform-iac-data-engineering提交8e7bd5f

许可证： 无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 reason-machines/data-skills 的技能

Youtube Channel Analytics Dashboard

reason-machines

介绍一款用于跟踪 YouTube 频道观看量、订阅者、收入和受众数据的 Windows 仪表板工具。

Data & Analytics42个月前更新

Terraform Data Engineering Infrastructure

reason-machines

使用 Terraform 为数据工程配置 AWS 基础设施（S3、EC2、IAM）的模式与命令。

DevOps & Cloud42个月前更新

Retail Etl Pipeline Medallion

reason-machines

指导使用 SQL Server 和可选 PySpark 构建包含青铜、白银、黄金三层的零售 ETL 数据仓库。

Data & Analytics42个月前更新

Power Bi Salespulse 360 Dashboard

reason-machines

指导构建 SalesPulse 360 Power BI 零售仪表板，用于销售、利润、预测和预警分析。

Data & Analytics42个月前更新

Logifleet Pulse Supply Chain Analytics

reason-machines

在 MS SQL Server 上构建物流数据仓库，并用 Power BI 仪表板分析仓储与车队数据。

Data & Analytics42个月前更新

Infrastructure Cicd Data Engineering

reason-machines

使用 Terraform 和 GitHub Actions 通过 OIDC 搭建部署 AWS 数据基础设施的 CI/CD 流水线。

DevOps & Cloud42个月前更新

更多DevOps & Cloud技能

Playwright Devops

microsoft

精选

面向 Playwright 的 DevOps 工作流：分析 main 分支最新提交的 GitHub Actions 失败并下载失败作业日志。

包含脚本
DevOps & Cloud97K今天更新

M5 Onboard

anthropics

精选

通过 USB 检测 M5Stack ESP32 开发板，刷写 UIFlow 2.0 固件并安装 MicroPython 应用包。

DevOps & Cloud37K今天更新

Yeet

openai

精选

使用 GitHub CLI 一次性完成暂存、提交、推送并创建或更新 GitHub 拉取请求。

DevOps & Cloud27K4周前更新

Runbook

anthropics

精选

为重复性任务创建或更新分步运维手册，包含故障排查、回滚和升级流程。

DevOps & Cloud27K今天更新

Incident Response

anthropics

精选

指导事故响应流程：严重级别判定、状态更新、缓解跟踪以及无责事后复盘。

DevOps & Cloud27K今天更新

Deploy Checklist

anthropics

精选

生成部署前就绪检查清单，涵盖部署前、部署、部署后和回滚触发条件。

DevOps & Cloud27K今天更新