Terraform Data Engineering Infrastructure

reason-machines/data-skills/skills/terraform-data-engineering-infrastructure

作者 reason-machines8e7bd5f22574无许可证4 个星标收录于 2026年10月8日更新于 2026年10月8日仓库2个月前更新

Infrastructure-as-Code patterns for data engineering using Terraform to provision AWS resources (S3, EC2, IAM)

仅含说明DevOps & Cloud
AI 生成的概览

使用 Terraform 为数据工程配置 AWS 基础设施（S3、EC2、IAM）的模式与命令。

功能
该技能为数据工程团队提供使用 Terraform 配置 AWS 资源的基础设施即代码指导。内容涵盖用于数据湖的 S3 存储桶、用于处理的 EC2 实例、IAM 角色与策略、变量、输出、远程状态以及多环境布局。它还列出用于 init、plan、apply、查看、导入和销毁的 Terraform 与 AWS CLI 命令，并包含故障排查和最佳实践。
适用场景
适用于使用 Terraform 搭建或管理数据管道与分析工作负载的 AWS 基础设施时。适合配置数据湖存储、处理实例和访问策略，或复现开发、预发布和生产环境。
运行要求
需要一个具有 S3、EC2 和 IAM 权限及凭证的 AWS 账户、Terraform v1.0 或更高版本，以及已配置的 AWS CLI。需要访问 AWS 的网络连接。该技能不附带脚本，仅为说明和配置示例。

Terraform Data Engineering Infrastructure

Skill by ara.so — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) patterns for data engineering teams using Terraform to provision and manage AWS resources. It demonstrates how to automate the creation of data infrastructure including S3 buckets for data lakes, EC2 instances for processing, and IAM policies for secure access.

What This Project Does

  • Provisions AWS infrastructure specifically designed for data engineering workloads
  • Manages S3 buckets for data storage and data lake architectures
  • Creates EC2 instances for data processing and ETL jobs
  • Configures IAM roles and policies for secure resource access
  • Provides declarative infrastructure definitions that can be version-controlled
  • Enables reproducible environment creation across dev/staging/prod

Prerequisites

Before using this project, ensure you have:

  1. An AWS account with root or administrative access
  2. Terraform installed (v1.0+)
  3. AWS CLI installed and configured
  4. IAM user with appropriate permissions (S3, EC2, IAM full access)

Installing Prerequisites

bash
# Install Terraform (macOS)brew tap hashicorp/tapbrew install hashicorp/tap/terraform
# Install AWS CLI (macOS)brew install awscli
# Configure AWS CLIaws configure# Enter your AWS Access Key ID, Secret Access Key, region, and output format

Setting Up IAM Permissions

Create an IAM user with the following permissions for Terraform:

  • Full S3 access (AmazonS3FullAccess)
  • Full EC2 access (AmazonEC2FullAccess)
  • Full IAM access (IAMFullAccess)

Note: This is for development/learning. In production, use least-privilege policies.

bash
# Create access keys for your IAM useraws iam create-access-key --user-name your-terraform-user
# Configure AWS CLI with these credentialsaws configure --profile terraform

Project Structure

terraform/├── main.tf           # Main infrastructure definitions├── variables.tf      # Input variables (if present)├── outputs.tf        # Output values (if present)└── terraform.tfstate # State file (generated)

Key Terraform Commands

Initialize Terraform

bash
# Initialize the working directoryterraform -chdir=terraform init
# Validate configuration filesterraform -chdir=terraform validate
# Format configuration filesterraform -chdir=terraform fmt

Plan and Apply Infrastructure

bash
# Preview changes without applyingterraform -chdir=terraform plan
# Apply changes and create infrastructureterraform -chdir=terraform apply
# Apply without confirmation promptterraform -chdir=terraform apply -auto-approve

Inspect Infrastructure

bash
# List all resources in stateterraform -chdir=terraform state list
# Show details of a specific resourceterraform -chdir=terraform state show aws_s3_bucket.data_bucket
# Output current stateterraform -chdir=terraform show

Destroy Infrastructure

bash
# Destroy all managed infrastructureterraform -chdir=terraform destroy
# Destroy specific resourcesterraform -chdir=terraform destroy -target=aws_instance.data_processor

Configuration Patterns

Basic S3 Bucket for Data Lake

hcl
# terraform/main.tfterraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = "us-east-1"}
# S3 bucket for raw dataresource "aws_s3_bucket" "raw_data" {  bucket = "my-unique-raw-data-bucket-12345"    tags = {    Environment = "dev"    Purpose     = "data-lake-raw"    ManagedBy   = "terraform"  }}
# Enable versioning for data recoveryresource "aws_s3_bucket_versioning" "raw_data_versioning" {  bucket = aws_s3_bucket.raw_data.id    versioning_configuration {    status = "Enabled"  }}
# Block public accessresource "aws_s3_bucket_public_access_block" "raw_data_public_access" {  bucket = aws_s3_bucket.raw_data.id
  block_public_acls       = true  block_public_policy     = true  ignore_public_acls      = true  restrict_public_buckets = true}

EC2 Instance for Data Processing

hcl
# Security group for EC2 instanceresource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing EC2 instances"
  ingress {    description = "SSH access"    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["0.0.0.0/0"]  # Restrict this in production  }
  egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }
  tags = {    Name      = "data-processor-sg"    ManagedBy = "terraform"  }}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2 AMI (update for your region)  instance_type = "t3.medium"    vpc_security_group_ids = [aws_security_group.data_processor_sg.id]    iam_instance_profile = aws_iam_instance_profile.data_processor_profile.name
  user_data = <<-EOF              #!/bin/bash              yum update -y              yum install -y python3 python3-pip              pip3 install boto3 pandas              EOF
  tags = {    Name        = "data-processor"    Environment = "dev"    ManagedBy   = "terraform"  }}

IAM Role for EC2 to Access S3

hcl
# IAM role for EC2 instancesresource "aws_iam_role" "data_processor_role" {  name = "data-processor-role"
  assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })
  tags = {    ManagedBy = "terraform"  }}
# Policy to allow S3 accessresource "aws_iam_role_policy" "s3_access_policy" {  name = "s3-access-policy"  role = aws_iam_role.data_processor_role.id
  policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Action = [          "s3:GetObject",          "s3:PutObject",          "s3:ListBucket"        ]        Resource = [          aws_s3_bucket.raw_data.arn,          "${aws_s3_bucket.raw_data.arn}/*"        ]      }    ]  })}
# Instance profile for EC2resource "aws_iam_instance_profile" "data_processor_profile" {  name = "data-processor-profile"  role = aws_iam_role.data_processor_role.name}

Multi-Environment Setup with Variables

hcl
# terraform/variables.tfvariable "environment" {  description = "Environment name (dev, staging, prod)"  type        = string  default     = "dev"}
variable "bucket_prefix" {  description = "Prefix for S3 bucket names"  type        = string}
variable "instance_type" {  description = "EC2 instance type"  type        = string  default     = "t3.medium"}
variable "aws_region" {  description = "AWS region"  type        = string  default     = "us-east-1"}
# terraform/main.tfresource "aws_s3_bucket" "data_bucket" {  bucket = "${var.bucket_prefix}-${var.environment}-data"    tags = {    Environment = var.environment    ManagedBy   = "terraform"  }}

Apply with variables:

bash
terraform -chdir=terraform apply \  -var="environment=prod" \  -var="bucket_prefix=mycompany" \  -var="instance_type=t3.large"

Output Values for Integration

hcl
# terraform/outputs.tfoutput "s3_bucket_name" {  description = "Name of the S3 bucket"  value       = aws_s3_bucket.raw_data.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.raw_data.arn}
output "ec2_instance_id" {  description = "ID of the EC2 instance"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of the EC2 instance"  value       = aws_instance.data_processor.public_ip}
# View outputs# terraform -chdir=terraform output

Common Workflows

Initial Setup

bash
# Clone the repositorygit clone https://github.com/josephmachado/iac-for-data-engineering-terraform-.gitcd iac-for-data-engineering-terraform-
# Update bucket name in terraform/main.tf to be globally unique# Edit terraform/main.tf and change bucket name
# Initialize and applyterraform -chdir=terraform initterraform -chdir=terraform validateterraform -chdir=terraform fmtterraform -chdir=terraform apply

Verify Resources Created

bash
# List S3 bucketsaws s3 ls
# Check EC2 instancesaws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress}' \  --output table
# View Terraform stateterraform -chdir=terraform state listcat terraform/terraform.tfstate | jq -r '.resources[] | [.type, .name] | join(",")'

Update Infrastructure

bash
# Edit terraform files# Then preview changesterraform -chdir=terraform plan
# Apply changesterraform -chdir=terraform apply

Clean Up

bash
# Destroy all resourcesterraform -chdir=terraform destroy
# Verify cleanupaws s3 lsaws ec2 describe-instances --filters "Name=instance-state-name,Values=running"

Advanced Patterns

Data Lake Structure with Multiple Buckets

hcl
# Raw data bucketresource "aws_s3_bucket" "raw" {  bucket = "${var.bucket_prefix}-raw-${var.environment}"  tags = {    Layer = "raw"  }}
# Processed data bucketresource "aws_s3_bucket" "processed" {  bucket = "${var.bucket_prefix}-processed-${var.environment}"  tags = {    Layer = "processed"  }}
# Curated data bucketresource "aws_s3_bucket" "curated" {  bucket = "${var.bucket_prefix}-curated-${var.environment}"  tags = {    Layer = "curated"  }}
# Lifecycle policy for raw dataresource "aws_s3_bucket_lifecycle_configuration" "raw_lifecycle" {  bucket = aws_s3_bucket.raw.id
  rule {    id     = "archive-old-data"    status = "Enabled"
    transition {      days          = 90      storage_class = "GLACIER"    }
    expiration {      days = 365    }  }}

Remote State Management

hcl
# Create S3 bucket for stateresource "aws_s3_bucket" "terraform_state" {  bucket = "my-terraform-state-bucket-12345"    lifecycle {    prevent_destroy = true  }}
resource "aws_s3_bucket_versioning" "terraform_state_versioning" {  bucket = aws_s3_bucket.terraform_state.id    versioning_configuration {    status = "Enabled"  }}
# Configure backend (in a separate backend.tf file)# terraform {#   backend "s3" {#     bucket = "my-terraform-state-bucket-12345"#     key    = "data-engineering/terraform.tfstate"#     region = "us-east-1"#   }# }

Troubleshooting

Bucket Name Already Exists

Error: BucketAlreadyExists: The requested bucket name is not available

Solution: S3 bucket names must be globally unique. Change the bucket name in main.tf:

hcl
resource "aws_s3_bucket" "data_bucket" {  bucket = "your-unique-prefix-data-bucket-${random_id.bucket_suffix.hex}"}
resource "random_id" "bucket_suffix" {  byte_length = 4}

Insufficient IAM Permissions

Error: UnauthorizedOperation or AccessDenied

Solution: Verify IAM user has required permissions:

bash
# Check current user identityaws sts get-caller-identity
# Verify policies attached to useraws iam list-attached-user-policies --user-name your-terraform-user

State Lock Issues

Error: Error acquiring the state lock

Solution:

bash
# Force unlock (use with caution)terraform -chdir=terraform force-unlock LOCK_ID
# Or remove local state lock filerm terraform/.terraform.tfstate.lock.info

Resource Already Exists

Error: Resource already exists but not in state

Solution: Import existing resource:

bash
# Import S3 bucketterraform -chdir=terraform import aws_s3_bucket.data_bucket my-existing-bucket-name
# Import EC2 instanceterraform -chdir=terraform import aws_instance.data_processor i-1234567890abcdef0

Terraform State Drift

Error: Resources differ from state

Solution:

bash
# Refresh state to match real infrastructureterraform -chdir=terraform refresh
# Or during plan/applyterraform -chdir=terraform apply -refresh=true

Region-Specific AMI Issues

Error: Invalid AMI ID for region

Solution: Use data source to find correct AMI:

hcl
data "aws_ami" "amazon_linux_2" {  most_recent = true  owners      = ["amazon"]
  filter {    name   = "name"    values = ["amzn2-ami-hvm-*-x86_64-gp2"]  }}
resource "aws_instance" "data_processor" {  ami           = data.aws_ami.amazon_linux_2.id  instance_type = "t3.medium"}

Best Practices

  1. Use Remote State: Store Terraform state in S3 with versioning enabled
  2. Separate Environments: Use workspaces or separate state files for dev/staging/prod
  3. Least Privilege IAM: Use specific IAM policies instead of full access in production
  4. Tag Everything: Add consistent tags for cost tracking and resource management
  5. Version Control: Commit .tf files but exclude terraform.tfstate and .terraform/
  6. Plan Before Apply: Always run terraform plan before apply
  7. Use Variables: Parameterize configurations for reusability
  8. Enable Encryption: Use S3 bucket encryption and EBS encryption for EC2
  9. Implement Lifecycle Policies: Archive or delete old data automatically
  10. Document Dependencies: Use comments to explain resource relationships

Environment Variables

bash
# AWS credentials (preferred over hardcoding)export AWS_ACCESS_KEY_ID=your_access_keyexport AWS_SECRET_ACCESS_KEY=your_secret_keyexport AWS_DEFAULT_REGION=us-east-1
# Terraform variablesexport TF_VAR_environment=devexport TF_VAR_bucket_prefix=mycompanyexport TF_VAR_instance_type=t3.medium

来源与署名

来源：reason-machines/data-skills位于skills/terraform-data-engineering-infrastructure提交8e7bd5f

许可证： 无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 reason-machines/data-skills 的技能

Youtube Channel Analytics Dashboard

reason-machines

介绍一款用于跟踪 YouTube 频道观看量、订阅者、收入和受众数据的 Windows 仪表板工具。

Data & Analytics42个月前更新

Terraform Iac Data Engineering

reason-machines

提供用于在 AWS 上预置 S3、EC2 和 IAM 等数据工程资源的 Terraform 基础设施即代码模式。

DevOps & Cloud42个月前更新

Retail Etl Pipeline Medallion

reason-machines

指导使用 SQL Server 和可选 PySpark 构建包含青铜、白银、黄金三层的零售 ETL 数据仓库。

Data & Analytics42个月前更新

Power Bi Salespulse 360 Dashboard

reason-machines

指导构建 SalesPulse 360 Power BI 零售仪表板，用于销售、利润、预测和预警分析。

Data & Analytics42个月前更新

Logifleet Pulse Supply Chain Analytics

reason-machines

在 MS SQL Server 上构建物流数据仓库，并用 Power BI 仪表板分析仓储与车队数据。

Data & Analytics42个月前更新

Infrastructure Cicd Data Engineering

reason-machines

使用 Terraform 和 GitHub Actions 通过 OIDC 搭建部署 AWS 数据基础设施的 CI/CD 流水线。

DevOps & Cloud42个月前更新

更多DevOps & Cloud技能

Playwright Devops

microsoft

精选

面向 Playwright 的 DevOps 工作流：分析 main 分支最新提交的 GitHub Actions 失败并下载失败作业日志。

包含脚本
DevOps & Cloud97K今天更新

M5 Onboard

anthropics

精选

通过 USB 检测 M5Stack ESP32 开发板，刷写 UIFlow 2.0 固件并安装 MicroPython 应用包。

DevOps & Cloud37K今天更新

Yeet

openai

精选

使用 GitHub CLI 一次性完成暂存、提交、推送并创建或更新 GitHub 拉取请求。

DevOps & Cloud27K4周前更新

Runbook

anthropics

精选

为重复性任务创建或更新分步运维手册，包含故障排查、回滚和升级流程。

DevOps & Cloud27K今天更新

Incident Response

anthropics

精选

指导事故响应流程：严重级别判定、状态更新、缓解跟踪以及无责事后复盘。

DevOps & Cloud27K今天更新

Deploy Checklist

anthropics

精选

生成部署前就绪检查清单，涵盖部署前、部署、部署后和回滚触发条件。

DevOps & Cloud27K今天更新