Iac Terraform Data Engineering

reason-machines/data-skills/skills/iac-terraform-data-engineering

作者 reason-machines8e7bd5f22574無授權條款4 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫2 個月前更新

Infrastructure-as-Code fundamentals for data engineers using Terraform to provision AWS resources (S3, EC2, IAM)

僅含說明DevOps & Cloud
AI 產生的概覽

指導資料工程師使用 Terraform 基礎設施即程式碼來佈建 AWS S3、EC2 與 IAM 資源。

功能
此技能提供用於佈建 AWS 資料平台基礎設施的 Terraform 設定模式與指令，涵蓋 S3 儲存貯體、EC2 執行個體與 IAM 角色。它說明 init、validate、format、plan、apply 與 destroy 工作流程，以及狀態檢視與遠端 S3 後端設定。它也包含資料工程模式，例如具備版本控制與生命週期規則的資料湖儲存貯體、EC2 使用者資料初始化、輸出值，以及常見 Terraform 錯誤的疑難排解。
適用情境
適用於以程式碼方式搭配 Terraform 建置或管理 AWS 資料基礎設施的情境，例如為資料管線建立 S3 與 EC2 資源。也適合查閱 Terraform 狀態管理、遠端後端，或解決常見的 apply 與狀態鎖定錯誤。
執行需求
需要具備 AWS 帳戶並透過 AWS CLI 設定認證，同時安裝 Terraform CLI 與 AWS CLI。需要連線至 AWS 與 Terraform 提供者登錄檔的網路存取。此技能不附帶指令碼，僅包含說明與設定範例。

IaC for Data Engineering with Terraform

Skill by ara.so — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) templates and patterns for data engineers using Terraform to provision and manage AWS resources. It focuses on creating reproducible, version-controlled infrastructure for data platforms including S3 storage, EC2 compute instances, and IAM permissions.

What This Project Does

  • Provides Terraform configurations for common data engineering infrastructure on AWS
  • Demonstrates IaC best practices for S3 buckets, EC2 instances, and IAM roles
  • Shows state management and lifecycle operations for data infrastructure
  • Teaches reproducible infrastructure provisioning for data pipelines

Prerequisites

Before using this project, ensure you have:

  1. AWS Account with root or admin access
  2. Terraform CLI installed (installation guide)
  3. AWS CLI installed and configured (setup guide)
  4. AWS Credentials configured via aws configure

AWS IAM Setup

Create an IAM user with appropriate permissions:

  1. Create IAM User: Navigate to AWS Console → IAM → Users → Create user
  2. Create Inline Policy: Attach a custom policy to the user
  3. Grant Permissions: For development/learning, grant full access to:
    • Amazon S3
    • Amazon EC2
    • AWS IAM

⚠️ Security Note: Full service access is NOT recommended for production. Use least-privilege policies in production environments.

Project Structure

terraform/├── main.tf           # Main Terraform configuration├── variables.tf      # Input variables (if present)├── outputs.tf        # Output values (if present)└── terraform.tfstate # State file (generated)

Key Terraform Commands

Initialize Terraform

Initialize the working directory and download provider plugins:

bash
terraform -chdir=terraform init

Validate Configuration

Check if the configuration is syntactically valid:

bash
terraform -chdir=terraform validate

Format Code

Automatically format Terraform files to canonical style:

bash
terraform -chdir=terraform fmt

Plan Infrastructure Changes

Preview what Terraform will create/modify/destroy:

bash
terraform -chdir=terraform plan

Apply Configuration

Create or update infrastructure:

bash
terraform -chdir=terraform apply

Terraform will show a plan and ask for confirmation. Type yes to proceed.

Auto-approve (for automation)

bash
terraform -chdir=terraform apply -auto-approve

Destroy Infrastructure

Remove all resources managed by Terraform:

bash
terraform -chdir=terraform destroy

Configuration

Basic Terraform Configuration Example

Before applying, modify terraform/main.tf to customize resource names:

hcl
# terraform/main.tfterraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = "us-east-1"}
# S3 bucket for data storageresource "aws_s3_bucket" "data_bucket" {  bucket = "my-unique-data-engineering-bucket-12345"    tags = {    Name        = "Data Engineering Bucket"    Environment = "dev"    ManagedBy   = "Terraform"  }}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2  instance_type = "t2.micro"    tags = {    Name        = "Data Processor"    Environment = "dev"    ManagedBy   = "Terraform"  }}
# IAM role for EC2 instanceresource "aws_iam_role" "ec2_s3_role" {  name = "ec2-s3-access-role"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })}

Variables Configuration

Create terraform/variables.tf for reusable configurations:

hcl
variable "aws_region" {  description = "AWS region for resources"  type        = string  default     = "us-east-1"}
variable "environment" {  description = "Environment name"  type        = string  default     = "dev"}
variable "bucket_name" {  description = "S3 bucket name for data storage"  type        = string  # Set via terraform.tfvars or -var flag}

Use variables in main.tf:

hcl
provider "aws" {  region = var.aws_region}
resource "aws_s3_bucket" "data_bucket" {  bucket = var.bucket_name    tags = {    Environment = var.environment  }}

Create terraform/terraform.tfvars:

hcl
bucket_name  = "my-unique-bucket-name-2026"aws_region   = "us-west-2"environment  = "production"

State Management

Inspect State

List all resources in the state:

bash
terraform -chdir=terraform state list

View detailed state information:

bash
cat terraform/terraform.tfstate | jq -r '.resources[] | [.type, .name] | join(",")'

Remote State (Production Pattern)

For production, store state remotely in S3:

hcl
# terraform/backend.tfterraform {  backend "s3" {    bucket         = "my-terraform-state-bucket"    key            = "data-platform/terraform.tfstate"    region         = "us-east-1"    encrypt        = true    dynamodb_table = "terraform-state-lock"  }}

Initialize with backend configuration:

bash
terraform -chdir=terraform init -backend-config="bucket=${TERRAFORM_STATE_BUCKET}"

Verification Commands

Verify S3 Bucket Creation

bash
aws s3 ls

Verify EC2 Instance

bash
aws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress, PrivateIP:PrivateIpAddress}' \  --output table

Check Specific Resource

bash
terraform -chdir=terraform show aws_s3_bucket.data_bucket

Common Patterns for Data Engineering

Pattern 1: Data Lake with Multiple Buckets

hcl
# Raw data bucketresource "aws_s3_bucket" "raw_data" {  bucket = "my-data-lake-raw-${var.environment}"}
# Processed data bucketresource "aws_s3_bucket" "processed_data" {  bucket = "my-data-lake-processed-${var.environment}"}
# Enable versioning for data lineageresource "aws_s3_bucket_versioning" "raw_data_versioning" {  bucket = aws_s3_bucket.raw_data.id    versioning_configuration {    status = "Enabled"  }}
# Lifecycle rules for cost optimizationresource "aws_s3_bucket_lifecycle_configuration" "raw_data_lifecycle" {  bucket = aws_s3_bucket.raw_data.id    rule {    id     = "archive-old-data"    status = "Enabled"        transition {      days          = 90      storage_class = "GLACIER"    }  }}

Pattern 2: EC2 with Data Processing Tools

hcl
# Security group for data processorresource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing instances"    ingress {    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["0.0.0.0/0"]  # Restrict in production  }    egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }}
# EC2 instance with user data for setupresource "aws_instance" "data_processor" {  ami           = var.ami_id  instance_type = "t3.medium"    vpc_security_group_ids = [aws_security_group.data_processor_sg.id]  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name    user_data = <<-EOF              #!/bin/bash              yum update -y              yum install -y python3 python3-pip              pip3 install pandas boto3              EOF    tags = {    Name = "Data Processor Instance"  }}
# IAM instance profileresource "aws_iam_instance_profile" "ec2_profile" {  name = "ec2-data-processor-profile"  role = aws_iam_role.ec2_s3_role.name}

Pattern 3: Outputs for Integration

hcl
# terraform/outputs.tfoutput "s3_bucket_name" {  description = "Name of the S3 bucket"  value       = aws_s3_bucket.data_bucket.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.data_bucket.arn}
output "ec2_instance_id" {  description = "ID of the EC2 instance"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of the EC2 instance"  value       = aws_instance.data_processor.public_ip}

Access outputs:

bash
terraform -chdir=terraform outputterraform -chdir=terraform output -json | jq -r '.s3_bucket_name.value'

Troubleshooting

Issue: "Error acquiring the state lock"

Cause: Another Terraform process is running or a previous run didn't release the lock.

Solution:

bash
# Force unlock (use with caution)terraform -chdir=terraform force-unlock <LOCK_ID>

Issue: "bucket name already exists"

Cause: S3 bucket names must be globally unique across all AWS accounts.

Solution: Change the bucket name in main.tf to something unique:

hcl
resource "aws_s3_bucket" "data_bucket" {  bucket = "my-unique-name-${random_id.bucket_suffix.hex}"}
resource "random_id" "bucket_suffix" {  byte_length = 4}

Issue: "insufficient IAM permissions"

Cause: The IAM user doesn't have required permissions.

Solution: Verify IAM policy includes necessary actions:

json
{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:*",        "ec2:*",        "iam:*"      ],      "Resource": "*"    }  ]}

Issue: State file out of sync

Cause: Manual changes made outside Terraform.

Solution: Refresh the state:

bash
terraform -chdir=terraform refresh

Or import existing resources:

bash
terraform -chdir=terraform import aws_s3_bucket.data_bucket my-existing-bucket

Workflow Example

Complete workflow for setting up data infrastructure:

bash
# 1. Configure AWS credentialsexport AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"export AWS_DEFAULT_REGION="us-east-1"
# 2. Customize configurationcd terraform# Edit main.tf to set unique bucket name
# 3. Initialize Terraformterraform init
# 4. Validate configurationterraform validate
# 5. Format codeterraform fmt
# 6. Preview changesterraform plan
# 7. Apply configurationterraform apply
# 8. Verify resourcesaws s3 lsaws ec2 describe-instances --output table
# 9. When done, clean upterraform destroy

Best Practices for Data Engineering IaC

  1. Use variables for environment-specific values
  2. Enable S3 versioning for data lineage and recovery
  3. Tag all resources for cost tracking and management
  4. Store state remotely in S3 with encryption and locking
  5. Use modules to organize reusable infrastructure components
  6. Never commit .tfstate files or AWS credentials to version control
  7. Implement lifecycle rules on S3 for cost optimization
  8. Use IAM roles instead of access keys for EC2 instances
  9. Plan before apply to review changes
  10. Destroy unused resources to avoid unnecessary costs

來源與署名

來源：reason-machines/data-skills位於skills/iac-terraform-data-engineering提交8e7bd5f

授權條款： 無授權條款

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 reason-machines/data-skills 的技能

Youtube Channel Analytics Dashboard

reason-machines

介紹一款用於追蹤 YouTube 頻道觀看數、訂閱者、收益與受眾資料的 Windows 儀表板工具。

Data & Analytics42 個月前更新

Terraform Iac Data Engineering

reason-machines

提供用於在 AWS 上佈建 S3、EC2 和 IAM 等資料工程資源的 Terraform 基礎設施即程式碼模式。

DevOps & Cloud42 個月前更新

Terraform Data Engineering Infrastructure

reason-machines

使用 Terraform 為資料工程佈建 AWS 基礎設施（S3、EC2、IAM）的模式與指令。

DevOps & Cloud42 個月前更新

Retail Etl Pipeline Medallion

reason-machines

指導使用 SQL Server 與選用 PySpark 建立包含青銅、白銀、黃金三層的零售 ETL 資料倉儲。

Data & Analytics42 個月前更新

Power Bi Salespulse 360 Dashboard

reason-machines

指導建置 SalesPulse 360 Power BI 零售儀表板，用於銷售、利潤、預測與警示分析。

Data & Analytics42 個月前更新

Logifleet Pulse Supply Chain Analytics

reason-machines

在 MS SQL Server 上建置物流資料倉儲，並以 Power BI 儀表板分析倉儲與車隊資料。

Data & Analytics42 個月前更新

更多DevOps & Cloud技能

Playwright Devops

microsoft

精選

Playwright 的 DevOps 工作流程：分析 main 分支最新提交的 GitHub Actions 失敗並下載失敗作業日誌。

包含腳本
DevOps & Cloud97K今天更新

M5 Onboard

anthropics

精選

透過 USB 偵測 M5Stack ESP32 開發板，燒錄 UIFlow 2.0 韌體並安裝 MicroPython 應用程式套件。

DevOps & Cloud37K今天更新

Yeet

openai

精選

使用 GitHub CLI 一次完成暫存、提交、推送並建立或更新 GitHub 拉取請求。

DevOps & Cloud27K4 週前更新

Runbook

anthropics

精選

為重複性任務建立或更新逐步操作手冊，包含疑難排解、回復與升級流程。

DevOps & Cloud27K今天更新

Incident Response

anthropics

精選

引導事故回應流程：嚴重程度分級、狀態更新、緩解追蹤，以及無責事後檢討。

DevOps & Cloud27K今天更新

Deploy Checklist

anthropics

精選

產生部署前就緒檢查清單，涵蓋部署前、部署、部署後與回復觸發條件。

DevOps & Cloud27K今天更新