Iac Terraform Data Engineering

reason-machines/data-skills/skills/iac-terraform-data-engineering

作者 reason-machines8e7bd5f22574无许可证4 个星标收录于 2026年10月8日更新于 2026年10月8日仓库2个月前更新

Infrastructure-as-Code fundamentals for data engineers using Terraform to provision AWS resources (S3, EC2, IAM)

仅含说明DevOps & Cloud

AI 生成的概览

指导数据工程师使用 Terraform 基础设施即代码来配置 AWS S3、EC2 和 IAM 资源。

功能: 该技能提供用于配置 AWS 数据平台基础设施的 Terraform 配置模式与命令，涵盖 S3 存储桶、EC2 实例和 IAM 角色。它讲解 init、validate、format、plan、apply 和 destroy 工作流，以及状态查看和远程 S3 后端设置。它还包含数据工程模式，例如带版本控制和生命周期规则的数据湖存储桶、EC2 用户数据初始化、输出值，以及常见 Terraform 错误的排查方法。
适用场景: 适用于使用 Terraform 以代码方式搭建或管理 AWS 数据基础设施的场景，例如为数据管道创建 S3 和 EC2 资源。也适合查阅 Terraform 状态管理、远程后端，或解决常见的 apply 与状态锁错误。
运行要求: 需要具备 AWS 账户并通过 AWS CLI 配置凭证，同时安装 Terraform CLI 和 AWS CLI。需要访问 AWS 和 Terraform 提供程序注册表的网络连接。该技能不附带脚本，仅包含说明和配置示例。

IaC for Data Engineering with Terraform

Skill by ara.so — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) templates and patterns for data engineers using Terraform to provision and manage AWS resources. It focuses on creating reproducible, version-controlled infrastructure for data platforms including S3 storage, EC2 compute instances, and IAM permissions.

What This Project Does

Provides Terraform configurations for common data engineering infrastructure on AWS
Demonstrates IaC best practices for S3 buckets, EC2 instances, and IAM roles
Shows state management and lifecycle operations for data infrastructure
Teaches reproducible infrastructure provisioning for data pipelines

Prerequisites

Before using this project, ensure you have:

AWS Account with root or admin access
Terraform CLI installed (installation guide)
AWS CLI installed and configured (setup guide)
AWS Credentials configured via aws configure

AWS IAM Setup

Create an IAM user with appropriate permissions:

Create IAM User: Navigate to AWS Console → IAM → Users → Create user
Create Inline Policy: Attach a custom policy to the user
Grant Permissions: For development/learning, grant full access to:
- Amazon S3
- Amazon EC2
- AWS IAM

⚠️ Security Note: Full service access is NOT recommended for production. Use least-privilege policies in production environments.

Project Structure

terraform/├── main.tf           # Main Terraform configuration├── variables.tf      # Input variables (if present)├── outputs.tf        # Output values (if present)└── terraform.tfstate # State file (generated)

terraform/├── main.tf           # Main Terraform configuration├── variables.tf      # Input variables (if present)├── outputs.tf        # Output values (if present)└── terraform.tfstate # State file (generated)

Key Terraform Commands

Initialize Terraform

Initialize the working directory and download provider plugins:

Validate Configuration

Check if the configuration is syntactically valid:

Format Code

Automatically format Terraform files to canonical style:

Plan Infrastructure Changes

Preview what Terraform will create/modify/destroy:

Apply Configuration

Create or update infrastructure:

Terraform will show a plan and ask for confirmation. Type yes to proceed.

Auto-approve (for automation)

Destroy Infrastructure

Remove all resources managed by Terraform:

Configuration

Basic Terraform Configuration Example

Before applying, modify terraform/main.tf to customize resource names:

hcl

# terraform/main.tfterraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = "us-east-1"}
# S3 bucket for data storageresource "aws_s3_bucket" "data_bucket" {  bucket = "my-unique-data-engineering-bucket-12345"    tags = {    Name        = "Data Engineering Bucket"    Environment = "dev"    ManagedBy   = "Terraform"  }}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2  instance_type = "t2.micro"    tags = {    Name        = "Data Processor"    Environment = "dev"    ManagedBy   = "Terraform"  }}
# IAM role for EC2 instanceresource "aws_iam_role" "ec2_s3_role" {  name = "ec2-s3-access-role"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })}

# terraform/main.tfterraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = "us-east-1"}
# S3 bucket for data storageresource "aws_s3_bucket" "data_bucket" {  bucket = "my-unique-data-engineering-bucket-12345"    tags = {    Name        = "Data Engineering Bucket"    Environment = "dev"    ManagedBy   = "Terraform"  }}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2  instance_type = "t2.micro"    tags = {    Name        = "Data Processor"    Environment = "dev"    ManagedBy   = "Terraform"  }}
# IAM role for EC2 instanceresource "aws_iam_role" "ec2_s3_role" {  name = "ec2-s3-access-role"    assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })}

Variables Configuration

Create terraform/variables.tf for reusable configurations:

hcl

variable "aws_region" {  description = "AWS region for resources"  type        = string  default     = "us-east-1"}
variable "environment" {  description = "Environment name"  type        = string  default     = "dev"}
variable "bucket_name" {  description = "S3 bucket name for data storage"  type        = string  # Set via terraform.tfvars or -var flag}

variable "aws_region" {  description = "AWS region for resources"  type        = string  default     = "us-east-1"}
variable "environment" {  description = "Environment name"  type        = string  default     = "dev"}
variable "bucket_name" {  description = "S3 bucket name for data storage"  type        = string  # Set via terraform.tfvars or -var flag}

Use variables in main.tf:

hcl

provider "aws" {  region = var.aws_region}
resource "aws_s3_bucket" "data_bucket" {  bucket = var.bucket_name    tags = {    Environment = var.environment  }}

provider "aws" {  region = var.aws_region}
resource "aws_s3_bucket" "data_bucket" {  bucket = var.bucket_name    tags = {    Environment = var.environment  }}

Create terraform/terraform.tfvars:

State Management

Inspect State

List all resources in the state:

View detailed state information:

Remote State (Production Pattern)

For production, store state remotely in S3:

hcl

# terraform/backend.tfterraform {  backend "s3" {    bucket         = "my-terraform-state-bucket"    key            = "data-platform/terraform.tfstate"    region         = "us-east-1"    encrypt        = true    dynamodb_table = "terraform-state-lock"  }}

# terraform/backend.tfterraform {  backend "s3" {    bucket         = "my-terraform-state-bucket"    key            = "data-platform/terraform.tfstate"    region         = "us-east-1"    encrypt        = true    dynamodb_table = "terraform-state-lock"  }}

Initialize with backend configuration:

Verification Commands

Verify S3 Bucket Creation

Verify EC2 Instance

bash

aws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress, PrivateIP:PrivateIpAddress}' \  --output table

aws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress, PrivateIP:PrivateIpAddress}' \  --output table

Check Specific Resource

Common Patterns for Data Engineering

Pattern 1: Data Lake with Multiple Buckets

hcl

# Raw data bucketresource "aws_s3_bucket" "raw_data" {  bucket = "my-data-lake-raw-${var.environment}"}
# Processed data bucketresource "aws_s3_bucket" "processed_data" {  bucket = "my-data-lake-processed-${var.environment}"}
# Enable versioning for data lineageresource "aws_s3_bucket_versioning" "raw_data_versioning" {  bucket = aws_s3_bucket.raw_data.id    versioning_configuration {    status = "Enabled"  }}
# Lifecycle rules for cost optimizationresource "aws_s3_bucket_lifecycle_configuration" "raw_data_lifecycle" {  bucket = aws_s3_bucket.raw_data.id    rule {    id     = "archive-old-data"    status = "Enabled"        transition {      days          = 90      storage_class = "GLACIER"    }  }}

# Raw data bucketresource "aws_s3_bucket" "raw_data" {  bucket = "my-data-lake-raw-${var.environment}"}
# Processed data bucketresource "aws_s3_bucket" "processed_data" {  bucket = "my-data-lake-processed-${var.environment}"}
# Enable versioning for data lineageresource "aws_s3_bucket_versioning" "raw_data_versioning" {  bucket = aws_s3_bucket.raw_data.id    versioning_configuration {    status = "Enabled"  }}
# Lifecycle rules for cost optimizationresource "aws_s3_bucket_lifecycle_configuration" "raw_data_lifecycle" {  bucket = aws_s3_bucket.raw_data.id    rule {    id     = "archive-old-data"    status = "Enabled"        transition {      days          = 90      storage_class = "GLACIER"    }  }}

Pattern 2: EC2 with Data Processing Tools

hcl

# Security group for data processorresource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing instances"    ingress {    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["0.0.0.0/0"]  # Restrict in production  }    egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }}
# EC2 instance with user data for setupresource "aws_instance" "data_processor" {  ami           = var.ami_id  instance_type = "t3.medium"    vpc_security_group_ids = [aws_security_group.data_processor_sg.id]  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name    user_data = <<-EOF              #!/bin/bash              yum update -y              yum install -y python3 python3-pip              pip3 install pandas boto3              EOF    tags = {    Name = "Data Processor Instance"  }}
# IAM instance profileresource "aws_iam_instance_profile" "ec2_profile" {  name = "ec2-data-processor-profile"  role = aws_iam_role.ec2_s3_role.name}

# Security group for data processorresource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing instances"    ingress {    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["0.0.0.0/0"]  # Restrict in production  }    egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }}
# EC2 instance with user data for setupresource "aws_instance" "data_processor" {  ami           = var.ami_id  instance_type = "t3.medium"    vpc_security_group_ids = [aws_security_group.data_processor_sg.id]  iam_instance_profile   = aws_iam_instance_profile.ec2_profile.name    user_data = <<-EOF              #!/bin/bash              yum update -y              yum install -y python3 python3-pip              pip3 install pandas boto3              EOF    tags = {    Name = "Data Processor Instance"  }}
# IAM instance profileresource "aws_iam_instance_profile" "ec2_profile" {  name = "ec2-data-processor-profile"  role = aws_iam_role.ec2_s3_role.name}

Pattern 3: Outputs for Integration

hcl

# terraform/outputs.tfoutput "s3_bucket_name" {  description = "Name of the S3 bucket"  value       = aws_s3_bucket.data_bucket.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.data_bucket.arn}
output "ec2_instance_id" {  description = "ID of the EC2 instance"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of the EC2 instance"  value       = aws_instance.data_processor.public_ip}

# terraform/outputs.tfoutput "s3_bucket_name" {  description = "Name of the S3 bucket"  value       = aws_s3_bucket.data_bucket.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.data_bucket.arn}
output "ec2_instance_id" {  description = "ID of the EC2 instance"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of the EC2 instance"  value       = aws_instance.data_processor.public_ip}

Access outputs:

bash

terraform -chdir=terraform outputterraform -chdir=terraform output -json | jq -r '.s3_bucket_name.value'

terraform -chdir=terraform outputterraform -chdir=terraform output -json | jq -r '.s3_bucket_name.value'

Troubleshooting

Issue: "Error acquiring the state lock"

Cause: Another Terraform process is running or a previous run didn't release the lock.

Solution:

Issue: "bucket name already exists"

Cause: S3 bucket names must be globally unique across all AWS accounts.

Solution: Change the bucket name in main.tf to something unique:

hcl

resource "aws_s3_bucket" "data_bucket" {  bucket = "my-unique-name-${random_id.bucket_suffix.hex}"}
resource "random_id" "bucket_suffix" {  byte_length = 4}

resource "aws_s3_bucket" "data_bucket" {  bucket = "my-unique-name-${random_id.bucket_suffix.hex}"}
resource "random_id" "bucket_suffix" {  byte_length = 4}

Issue: "insufficient IAM permissions"

Cause: The IAM user doesn't have required permissions.

Solution: Verify IAM policy includes necessary actions:

json

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:*",        "ec2:*",        "iam:*"      ],      "Resource": "*"    }  ]}

{  "Version": "2012-10-17",  "Statement": [    {      "Effect": "Allow",      "Action": [        "s3:*",        "ec2:*",        "iam:*"      ],      "Resource": "*"    }  ]}

Issue: State file out of sync

Cause: Manual changes made outside Terraform.

Solution: Refresh the state:

Or import existing resources:

Workflow Example

Complete workflow for setting up data infrastructure:

bash

# 1. Configure AWS credentialsexport AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"export AWS_DEFAULT_REGION="us-east-1"
# 2. Customize configurationcd terraform# Edit main.tf to set unique bucket name
# 3. Initialize Terraformterraform init
# 4. Validate configurationterraform validate
# 5. Format codeterraform fmt
# 6. Preview changesterraform plan
# 7. Apply configurationterraform apply
# 8. Verify resourcesaws s3 lsaws ec2 describe-instances --output table
# 9. When done, clean upterraform destroy

# 1. Configure AWS credentialsexport AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}"export AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}"export AWS_DEFAULT_REGION="us-east-1"
# 2. Customize configurationcd terraform# Edit main.tf to set unique bucket name
# 3. Initialize Terraformterraform init
# 4. Validate configurationterraform validate
# 5. Format codeterraform fmt
# 6. Preview changesterraform plan
# 7. Apply configurationterraform apply
# 8. Verify resourcesaws s3 lsaws ec2 describe-instances --output table
# 9. When done, clean upterraform destroy

Best Practices for Data Engineering IaC

Use variables for environment-specific values
Enable S3 versioning for data lineage and recovery
Tag all resources for cost tracking and management
Store state remotely in S3 with encryption and locking
Use modules to organize reusable infrastructure components
Never commit .tfstate files or AWS credentials to version control
Implement lifecycle rules on S3 for cost optimization
Use IAM roles instead of access keys for EC2 instances
Plan before apply to review changes
Destroy unused resources to avoid unnecessary costs

来源与署名

来源：reason-machines/data-skills位于skills/iac-terraform-data-engineering提交8e7bd5f

许可证：无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

Iac Terraform Data Engineering

IaC for Data Engineering with Terraform

What This Project Does

Prerequisites

AWS IAM Setup

Project Structure

Key Terraform Commands

Initialize Terraform

Validate Configuration

Format Code

Plan Infrastructure Changes

Apply Configuration

Auto-approve (for automation)

Destroy Infrastructure

Configuration

Basic Terraform Configuration Example

Variables Configuration

State Management

Inspect State

Remote State (Production Pattern)

Verification Commands

Verify S3 Bucket Creation

Verify EC2 Instance

Check Specific Resource

Common Patterns for Data Engineering

Pattern 1: Data Lake with Multiple Buckets

Pattern 2: EC2 with Data Processing Tools

Pattern 3: Outputs for Integration

Troubleshooting

Issue: "Error acquiring the state lock"

Issue: "bucket name already exists"

Issue: "insufficient IAM permissions"

Issue: State file out of sync

Workflow Example

Best Practices for Data Engineering IaC

来源与署名

更多来自 reason-machines/data-skills 的技能

Youtube Channel Analytics Dashboard

Terraform Iac Data Engineering

Terraform Data Engineering Infrastructure

Retail Etl Pipeline Medallion

Power Bi Salespulse 360 Dashboard

Logifleet Pulse Supply Chain Analytics

更多DevOps & Cloud技能

Playwright Devops

M5 Onboard

Yeet

Runbook

Incident Response

Deploy Checklist