Terraform Iac Data Engineering

reason-machines/data-skills/skills/terraform-iac-data-engineering

作者 reason-machines8e7bd5f22574無授權條款4 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫2 個月前更新

Infrastructure-as-Code with Terraform for data engineering on AWS (S3, EC2, IAM)

僅含說明DevOps & Cloud
AI 產生的概覽

提供用於在 AWS 上佈建 S3、EC2 和 IAM 等資料工程資源的 Terraform 基礎設施即程式碼模式。

功能
此技能提供使用 Terraform 為資料工程工作負載佈建 AWS 基礎設施的參考指引與設定範例。內容涵蓋 S3 儲存貯體、EC2 執行個體、IAM 角色與政策、VPC 與安全群組、遠端狀態後端、生命週期政策以及多環境模組結構。它也記錄了 init、plan、apply、狀態檢視、匯入、銷毀和疑難排解等 Terraform 指令。
適用情境
適用於使用 Terraform 建置或管理資料管線的 AWS 基礎設施，包括 S3 儲存、EC2 處理執行個體和 IAM 存取權限。也適合處理狀態管理、遠端後端和多環境基礎設施模式。
執行需求
需要 Terraform CLI 和 AWS CLI，並設定具有建立 S3、EC2、IAM、VPC 和 DynamoDB 資源權限的 AWS 憑證。需要連線至 AWS 和 Terraform 提供者登錄檔的網路存取。此技能僅為說明文件，不附帶指令碼。

Terraform IaC for Data Engineering

Skill by ara.so — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) patterns using Terraform specifically for data engineering workloads on AWS. It demonstrates how to provision and manage AWS resources (S3, EC2, IAM) needed for data pipelines and processing.

What This Project Does

  • Provisions AWS S3 buckets for data storage
  • Creates EC2 instances for data processing workloads
  • Manages IAM users, roles, and policies
  • Demonstrates Terraform state management
  • Provides reusable IaC patterns for data engineering infrastructure

Installation

Prerequisites

  1. Terraform CLI

    bash
    # macOSbrew install terraform
# Linuxwget https://releases.hashicorp.com/terraform/1.5.0/terraform_1.5.0_linux_amd64.zipunzip terraform_1.5.0_linux_amd64.zipsudo mv terraform /usr/local/bin/

  2. AWS CLI

    bash
    # macOSbrew install awscli
# Linuxcurl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"unzip awscliv2.zipsudo ./aws/install

  3. Configure AWS CLI

    bash
    aws configure# Enter your AWS Access Key ID# Enter your AWS Secret Access Key# Default region: us-east-1# Default output format: json

Project Setup

bash
git clone https://github.com/josephmachado/iac-for-data-engineering-terraform-.gitcd iac-for-data-engineering-terraform-

Key Terraform Commands

Initialize Terraform

bash
# Initialize terraform (downloads providers, sets up backend)terraform -chdir=terraform init
# Validate configuration filesterraform -chdir=terraform validate
# Format configuration filesterraform -chdir=terraform fmt

Plan and Apply Infrastructure

bash
# Preview changes before applyingterraform -chdir=terraform plan
# Apply infrastructure changesterraform -chdir=terraform apply
# Auto-approve without confirmation (use with caution)terraform -chdir=terraform apply -auto-approve

Inspect Infrastructure

bash
# List all resources in stateterraform -chdir=terraform state list
# Show details of a specific resourceterraform -chdir=terraform state show aws_s3_bucket.data_bucket
# Output specific valuesterraform -chdir=terraform output
# Show current state in JSONterraform -chdir=terraform show -json

Destroy Infrastructure

bash
# Destroy all managed infrastructureterraform -chdir=terraform destroy
# Destroy specific resourceterraform -chdir=terraform destroy -target=aws_instance.data_processor

Configuration Structure

Basic Terraform Configuration for Data Engineering

main.tf - Core infrastructure definition:

hcl
terraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = var.aws_region}
# S3 bucket for data storageresource "aws_s3_bucket" "data_lake" {  bucket = "my-unique-data-lake-bucket-${var.environment}"    tags = {    Name        = "Data Lake Bucket"    Environment = var.environment    Project     = "DataEngineering"  }}
# Enable versioning for data protectionresource "aws_s3_bucket_versioning" "data_lake_versioning" {  bucket = aws_s3_bucket.data_lake.id    versioning_configuration {    status = "Enabled"  }}
# Block public accessresource "aws_s3_bucket_public_access_block" "data_lake_public_access" {  bucket = aws_s3_bucket.data_lake.id    block_public_acls       = true  block_public_policy     = true  ignore_public_acls      = true  restrict_public_buckets = true}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = var.ec2_ami  instance_type = var.ec2_instance_type    tags = {    Name        = "DataProcessor"    Environment = var.environment  }    user_data = <<-EOF              #!/bin/bash              sudo yum update -y              sudo yum install -y python3 python3-pip              pip3 install pandas boto3              EOF}
# IAM role for EC2 to access S3resource "aws_iam_role" "ec2_s3_access_role" {  name = "ec2-s3-access-role"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })}
# IAM policy for S3 accessresource "aws_iam_role_policy" "ec2_s3_policy" {  name = "ec2-s3-policy"  role = aws_iam_role.ec2_s3_access_role.id    policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Action = [          "s3:GetObject",          "s3:PutObject",          "s3:ListBucket"        ]        Resource = [          aws_s3_bucket.data_lake.arn,          "${aws_s3_bucket.data_lake.arn}/*"        ]      }    ]  })}
# Attach IAM role to EC2 instanceresource "aws_iam_instance_profile" "ec2_profile" {  name = "ec2-s3-profile"  role = aws_iam_role.ec2_s3_access_role.name}

variables.tf - Input variables:

hcl
variable "aws_region" {  description = "AWS region for resources"  type        = string  default     = "us-east-1"}
variable "environment" {  description = "Environment name (dev, staging, prod)"  type        = string  default     = "dev"}
variable "ec2_ami" {  description = "AMI ID for EC2 instance"  type        = string  default     = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2}
variable "ec2_instance_type" {  description = "EC2 instance type"  type        = string  default     = "t2.micro"}
variable "bucket_prefix" {  description = "Prefix for S3 bucket names"  type        = string  default     = "data-eng"}

outputs.tf - Output values:

hcl
output "s3_bucket_name" {  description = "Name of the S3 data lake bucket"  value       = aws_s3_bucket.data_lake.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.data_lake.arn}
output "ec2_instance_id" {  description = "ID of the EC2 data processor"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of EC2 instance"  value       = aws_instance.data_processor.public_ip}

terraform.tfvars - Variable values (gitignore this file):

hcl
aws_region         = "us-west-2"environment        = "production"ec2_instance_type  = "t3.medium"bucket_prefix      = "my-company-data"

Common Data Engineering Patterns

Multi-Environment Setup

environments/dev/main.tf:

hcl
module "data_infrastructure" {  source = "../../modules/data-infra"    environment       = "dev"  instance_type     = "t2.micro"  enable_monitoring = false}

environments/prod/main.tf:

hcl
module "data_infrastructure" {  source = "../../modules/data-infra"    environment       = "prod"  instance_type     = "t3.xlarge"  enable_monitoring = true  backup_enabled    = true}

S3 Bucket with Lifecycle Policies

hcl
resource "aws_s3_bucket" "data_archive" {  bucket = "data-archive-${var.environment}"}
resource "aws_s3_bucket_lifecycle_configuration" "data_archive_lifecycle" {  bucket = aws_s3_bucket.data_archive.id    rule {    id     = "archive-old-data"    status = "Enabled"        transition {      days          = 30      storage_class = "STANDARD_IA"    }        transition {      days          = 90      storage_class = "GLACIER"    }        expiration {      days = 365    }  }    rule {    id     = "delete-incomplete-uploads"    status = "Enabled"        abort_incomplete_multipart_upload {      days_after_initiation = 7    }  }}

VPC Setup for Data Processing

hcl
resource "aws_vpc" "data_vpc" {  cidr_block           = "10.0.0.0/16"  enable_dns_hostnames = true  enable_dns_support   = true    tags = {    Name = "data-engineering-vpc"  }}
resource "aws_subnet" "private_subnet" {  vpc_id            = aws_vpc.data_vpc.id  cidr_block        = "10.0.1.0/24"  availability_zone = "${var.aws_region}a"    tags = {    Name = "private-data-subnet"  }}
resource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing instances"  vpc_id      = aws_vpc.data_vpc.id    egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }    ingress {    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["10.0.0.0/16"]  }}

Remote State Configuration

backend.tf:

hcl
terraform {  backend "s3" {    bucket         = "terraform-state-bucket-unique-name"    key            = "data-engineering/terraform.tfstate"    region         = "us-east-1"    encrypt        = true    dynamodb_table = "terraform-state-lock"  }}

Create state backend resources:

hcl
resource "aws_s3_bucket" "terraform_state" {  bucket = "terraform-state-bucket-unique-name"}
resource "aws_s3_bucket_versioning" "terraform_state_versioning" {  bucket = aws_s3_bucket.terraform_state.id    versioning_configuration {    status = "Enabled"  }}
resource "aws_dynamodb_table" "terraform_locks" {  name         = "terraform-state-lock"  billing_mode = "PAY_PER_REQUEST"  hash_key     = "LockID"    attribute {    name = "LockID"    type = "S"  }}

Verification and Testing

Verify S3 Bucket Creation

bash
# List all S3 bucketsaws s3 ls
# Check specific bucketaws s3 ls s3://my-unique-data-lake-bucket-dev/
# Upload test fileecho "test data" > test.txtaws s3 cp test.txt s3://my-unique-data-lake-bucket-dev/

Verify EC2 Instances

bash
# List running instancesaws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress}' \  --output table
# Get specific instance detailsaws ec2 describe-instances \  --instance-ids $(terraform -chdir=terraform output -raw ec2_instance_id)

Verify IAM Roles

bash
# List IAM rolesaws iam list-roles --query 'Roles[?contains(RoleName, `ec2-s3-access`)].RoleName'
# Get role policyaws iam get-role-policy \  --role-name ec2-s3-access-role \  --policy-name ec2-s3-policy

State Management

Inspect State

bash
# View state file (formatted)cat terraform/terraform.tfstate | jq -r '.resources[] | [.type, .name] | join(",")'
# List resources in stateterraform -chdir=terraform state list
# Show resource detailsterraform -chdir=terraform state show aws_s3_bucket.data_lake

Import Existing Resources

bash
# Import existing S3 bucketterraform -chdir=terraform import aws_s3_bucket.data_lake my-existing-bucket
# Import existing EC2 instanceterraform -chdir=terraform import aws_instance.data_processor i-1234567890abcdef0

Move Resources in State

bash
# Rename resource in stateterraform -chdir=terraform state mv aws_s3_bucket.old_name aws_s3_bucket.new_name

Troubleshooting

Common Issues

Issue: Bucket name already exists

hcl
# Solution: Use unique bucket name with random suffixresource "random_id" "bucket_suffix" {  byte_length = 4}
resource "aws_s3_bucket" "data_lake" {  bucket = "data-lake-${var.environment}-${random_id.bucket_suffix.hex}"}

Issue: AWS credentials not found

bash
# Check AWS configurationaws configure list
# Use environment variablesexport AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"export AWS_DEFAULT_REGION="us-east-1"

Issue: State file locked

bash
# Force unlock (use with caution)terraform -chdir=terraform force-unlock <LOCK_ID>

Issue: Resource already exists

bash
# Import existing resourceterraform -chdir=terraform import <resource_type>.<resource_name> <resource_id>
# Or remove from stateterraform -chdir=terraform state rm <resource_type>.<resource_name>

Issue: Terraform version mismatch

hcl
# Specify required version in terraform blockterraform {  required_version = ">= 1.5.0"    required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}

Debugging

bash
# Enable debug loggingexport TF_LOG=DEBUGterraform -chdir=terraform apply
# Log to fileexport TF_LOG_PATH=terraform-debug.logterraform -chdir=terraform apply
# Disable loggingunset TF_LOGunset TF_LOG_PATH

Validate and Format

bash
# Validate configurationterraform -chdir=terraform validate
# Format all filesterraform -chdir=terraform fmt -recursive
# Check formatting without making changesterraform -chdir=terraform fmt -check

Best Practices

  1. Always use variables for environment-specific values
  2. Enable S3 versioning for state files and data buckets
  3. Use remote state for team collaboration
  4. Tag all resources with environment, project, and owner
  5. Implement lifecycle policies for cost optimization
  6. Use modules for reusable infrastructure patterns
  7. Store secrets in AWS Secrets Manager, reference via data sources
  8. Run terraform plan before apply
  9. Use workspaces for multiple environments
  10. Document your infrastructure with comments and README files

來源與署名

來源：reason-machines/data-skills位於skills/terraform-iac-data-engineering提交8e7bd5f

授權條款： 無授權條款

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 reason-machines/data-skills 的技能

Youtube Channel Analytics Dashboard

reason-machines

介紹一款用於追蹤 YouTube 頻道觀看數、訂閱者、收益與受眾資料的 Windows 儀表板工具。

Data & Analytics42 個月前更新

Terraform Data Engineering Infrastructure

reason-machines

使用 Terraform 為資料工程佈建 AWS 基礎設施（S3、EC2、IAM）的模式與指令。

DevOps & Cloud42 個月前更新

Retail Etl Pipeline Medallion

reason-machines

指導使用 SQL Server 與選用 PySpark 建立包含青銅、白銀、黃金三層的零售 ETL 資料倉儲。

Data & Analytics42 個月前更新

Power Bi Salespulse 360 Dashboard

reason-machines

指導建置 SalesPulse 360 Power BI 零售儀表板，用於銷售、利潤、預測與警示分析。

Data & Analytics42 個月前更新

Logifleet Pulse Supply Chain Analytics

reason-machines

在 MS SQL Server 上建置物流資料倉儲，並以 Power BI 儀表板分析倉儲與車隊資料。

Data & Analytics42 個月前更新

Infrastructure Cicd Data Engineering

reason-machines

使用 Terraform 與 GitHub Actions 透過 OIDC 建立部署 AWS 資料基礎設施的 CI/CD 流程。

DevOps & Cloud42 個月前更新

更多DevOps & Cloud技能

Playwright Devops

microsoft

精選

Playwright 的 DevOps 工作流程：分析 main 分支最新提交的 GitHub Actions 失敗並下載失敗作業日誌。

包含腳本
DevOps & Cloud97K今天更新

M5 Onboard

anthropics

精選

透過 USB 偵測 M5Stack ESP32 開發板，燒錄 UIFlow 2.0 韌體並安裝 MicroPython 應用程式套件。

DevOps & Cloud37K今天更新

Yeet

openai

精選

使用 GitHub CLI 一次完成暫存、提交、推送並建立或更新 GitHub 拉取請求。

DevOps & Cloud27K4 週前更新

Runbook

anthropics

精選

為重複性任務建立或更新逐步操作手冊，包含疑難排解、回復與升級流程。

DevOps & Cloud27K今天更新

Incident Response

anthropics

精選

引導事故回應流程：嚴重程度分級、狀態更新、緩解追蹤，以及無責事後檢討。

DevOps & Cloud27K今天更新

Deploy Checklist

anthropics

精選

產生部署前就緒檢查清單，涵蓋部署前、部署、部署後與回復觸發條件。

DevOps & Cloud27K今天更新