Terraform Data Engineering Infrastructure

reason-machines/data-skills/skills/terraform-data-engineering-infrastructure

by reason-machines8e7bd5f22574No license4 starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 2 months ago

Infrastructure-as-Code patterns for data engineering using Terraform to provision AWS resources (S3, EC2, IAM)

Instructions onlyDevOps & Cloud

AI-generated overview

Terraform patterns and commands for provisioning AWS data engineering infrastructure such as S3, EC2 and IAM.

What it does: This skill supplies Infrastructure-as-Code guidance for data engineering teams using Terraform to provision AWS resources. It covers S3 buckets for data lakes, EC2 instances for processing, IAM roles and policies, variables, outputs, remote state and multi-environment layouts. It also lists Terraform and AWS CLI commands for init, plan, apply, inspect, import and destroy, plus troubleshooting and best practices.
When to use it: Use it when setting up or managing AWS infrastructure for data pipelines and analytics workloads with Terraform. It fits provisioning data lake storage, processing instances and access policies, or reproducing dev, staging and prod environments.
Requirements: Requires an AWS account with credentials and permissions for S3, EC2 and IAM, Terraform v1.0 or later, and the AWS CLI configured. Network access to AWS is needed. It ships no scripts; it is instructions and configuration examples only.

Terraform Data Engineering Infrastructure

Skill by ara.so — Data Skills collection.

This project provides Infrastructure-as-Code (IaC) patterns for data engineering teams using Terraform to provision and manage AWS resources. It demonstrates how to automate the creation of data infrastructure including S3 buckets for data lakes, EC2 instances for processing, and IAM policies for secure access.

What This Project Does

Provisions AWS infrastructure specifically designed for data engineering workloads
Manages S3 buckets for data storage and data lake architectures
Creates EC2 instances for data processing and ETL jobs
Configures IAM roles and policies for secure resource access
Provides declarative infrastructure definitions that can be version-controlled
Enables reproducible environment creation across dev/staging/prod

Prerequisites

Before using this project, ensure you have:

An AWS account with root or administrative access
Terraform installed (v1.0+)
AWS CLI installed and configured
IAM user with appropriate permissions (S3, EC2, IAM full access)

Installing Prerequisites

bash

# Install Terraform (macOS)brew tap hashicorp/tapbrew install hashicorp/tap/terraform
# Install AWS CLI (macOS)brew install awscli
# Configure AWS CLIaws configure# Enter your AWS Access Key ID, Secret Access Key, region, and output format

# Install Terraform (macOS)brew tap hashicorp/tapbrew install hashicorp/tap/terraform
# Install AWS CLI (macOS)brew install awscli
# Configure AWS CLIaws configure# Enter your AWS Access Key ID, Secret Access Key, region, and output format

Setting Up IAM Permissions

Create an IAM user with the following permissions for Terraform:

Full S3 access (AmazonS3FullAccess)
Full EC2 access (AmazonEC2FullAccess)
Full IAM access (IAMFullAccess)

Note: This is for development/learning. In production, use least-privilege policies.

bash

# Create access keys for your IAM useraws iam create-access-key --user-name your-terraform-user
# Configure AWS CLI with these credentialsaws configure --profile terraform

# Create access keys for your IAM useraws iam create-access-key --user-name your-terraform-user
# Configure AWS CLI with these credentialsaws configure --profile terraform

Project Structure

terraform/├── main.tf           # Main infrastructure definitions├── variables.tf      # Input variables (if present)├── outputs.tf        # Output values (if present)└── terraform.tfstate # State file (generated)

terraform/├── main.tf           # Main infrastructure definitions├── variables.tf      # Input variables (if present)├── outputs.tf        # Output values (if present)└── terraform.tfstate # State file (generated)

Key Terraform Commands

Initialize Terraform

bash

# Initialize the working directoryterraform -chdir=terraform init
# Validate configuration filesterraform -chdir=terraform validate
# Format configuration filesterraform -chdir=terraform fmt

# Initialize the working directoryterraform -chdir=terraform init
# Validate configuration filesterraform -chdir=terraform validate
# Format configuration filesterraform -chdir=terraform fmt

Plan and Apply Infrastructure

bash

# Preview changes without applyingterraform -chdir=terraform plan
# Apply changes and create infrastructureterraform -chdir=terraform apply
# Apply without confirmation promptterraform -chdir=terraform apply -auto-approve

# Preview changes without applyingterraform -chdir=terraform plan
# Apply changes and create infrastructureterraform -chdir=terraform apply
# Apply without confirmation promptterraform -chdir=terraform apply -auto-approve

Inspect Infrastructure

bash

# List all resources in stateterraform -chdir=terraform state list
# Show details of a specific resourceterraform -chdir=terraform state show aws_s3_bucket.data_bucket
# Output current stateterraform -chdir=terraform show

# List all resources in stateterraform -chdir=terraform state list
# Show details of a specific resourceterraform -chdir=terraform state show aws_s3_bucket.data_bucket
# Output current stateterraform -chdir=terraform show

Destroy Infrastructure

bash

# Destroy all managed infrastructureterraform -chdir=terraform destroy
# Destroy specific resourcesterraform -chdir=terraform destroy -target=aws_instance.data_processor

# Destroy all managed infrastructureterraform -chdir=terraform destroy
# Destroy specific resourcesterraform -chdir=terraform destroy -target=aws_instance.data_processor

Configuration Patterns

Basic S3 Bucket for Data Lake

hcl

# terraform/main.tfterraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = "us-east-1"}
# S3 bucket for raw dataresource "aws_s3_bucket" "raw_data" {  bucket = "my-unique-raw-data-bucket-12345"    tags = {    Environment = "dev"    Purpose     = "data-lake-raw"    ManagedBy   = "terraform"  }}
# Enable versioning for data recoveryresource "aws_s3_bucket_versioning" "raw_data_versioning" {  bucket = aws_s3_bucket.raw_data.id    versioning_configuration {    status = "Enabled"  }}
# Block public accessresource "aws_s3_bucket_public_access_block" "raw_data_public_access" {  bucket = aws_s3_bucket.raw_data.id
  block_public_acls       = true  block_public_policy     = true  ignore_public_acls      = true  restrict_public_buckets = true}

# terraform/main.tfterraform {  required_providers {    aws = {      source  = "hashicorp/aws"      version = "~> 5.0"    }  }}
provider "aws" {  region = "us-east-1"}
# S3 bucket for raw dataresource "aws_s3_bucket" "raw_data" {  bucket = "my-unique-raw-data-bucket-12345"    tags = {    Environment = "dev"    Purpose     = "data-lake-raw"    ManagedBy   = "terraform"  }}
# Enable versioning for data recoveryresource "aws_s3_bucket_versioning" "raw_data_versioning" {  bucket = aws_s3_bucket.raw_data.id    versioning_configuration {    status = "Enabled"  }}
# Block public accessresource "aws_s3_bucket_public_access_block" "raw_data_public_access" {  bucket = aws_s3_bucket.raw_data.id
  block_public_acls       = true  block_public_policy     = true  ignore_public_acls      = true  restrict_public_buckets = true}

EC2 Instance for Data Processing

hcl

# Security group for EC2 instanceresource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing EC2 instances"
  ingress {    description = "SSH access"    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["0.0.0.0/0"]  # Restrict this in production  }
  egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }
  tags = {    Name      = "data-processor-sg"    ManagedBy = "terraform"  }}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2 AMI (update for your region)  instance_type = "t3.medium"    vpc_security_group_ids = [aws_security_group.data_processor_sg.id]    iam_instance_profile = aws_iam_instance_profile.data_processor_profile.name
  user_data = <<-EOF              #!/bin/bash              yum update -y              yum install -y python3 python3-pip              pip3 install boto3 pandas              EOF
  tags = {    Name        = "data-processor"    Environment = "dev"    ManagedBy   = "terraform"  }}

# Security group for EC2 instanceresource "aws_security_group" "data_processor_sg" {  name        = "data-processor-sg"  description = "Security group for data processing EC2 instances"
  ingress {    description = "SSH access"    from_port   = 22    to_port     = 22    protocol    = "tcp"    cidr_blocks = ["0.0.0.0/0"]  # Restrict this in production  }
  egress {    from_port   = 0    to_port     = 0    protocol    = "-1"    cidr_blocks = ["0.0.0.0/0"]  }
  tags = {    Name      = "data-processor-sg"    ManagedBy = "terraform"  }}
# EC2 instance for data processingresource "aws_instance" "data_processor" {  ami           = "ami-0c55b159cbfafe1f0"  # Amazon Linux 2 AMI (update for your region)  instance_type = "t3.medium"    vpc_security_group_ids = [aws_security_group.data_processor_sg.id]    iam_instance_profile = aws_iam_instance_profile.data_processor_profile.name
  user_data = <<-EOF              #!/bin/bash              yum update -y              yum install -y python3 python3-pip              pip3 install boto3 pandas              EOF
  tags = {    Name        = "data-processor"    Environment = "dev"    ManagedBy   = "terraform"  }}

IAM Role for EC2 to Access S3

hcl

# IAM role for EC2 instancesresource "aws_iam_role" "data_processor_role" {  name = "data-processor-role"
  assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })
  tags = {    ManagedBy = "terraform"  }}
# Policy to allow S3 accessresource "aws_iam_role_policy" "s3_access_policy" {  name = "s3-access-policy"  role = aws_iam_role.data_processor_role.id
  policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Action = [          "s3:GetObject",          "s3:PutObject",          "s3:ListBucket"        ]        Resource = [          aws_s3_bucket.raw_data.arn,          "${aws_s3_bucket.raw_data.arn}/*"        ]      }    ]  })}
# Instance profile for EC2resource "aws_iam_instance_profile" "data_processor_profile" {  name = "data-processor-profile"  role = aws_iam_role.data_processor_role.name}

# IAM role for EC2 instancesresource "aws_iam_role" "data_processor_role" {  name = "data-processor-role"
  assume_role_policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Action = "sts:AssumeRole"        Effect = "Allow"        Principal = {          Service = "ec2.amazonaws.com"        }      }    ]  })
  tags = {    ManagedBy = "terraform"  }}
# Policy to allow S3 accessresource "aws_iam_role_policy" "s3_access_policy" {  name = "s3-access-policy"  role = aws_iam_role.data_processor_role.id
  policy = jsonencode({    Version = "2012-10-17"    Statement = [      {        Effect = "Allow"        Action = [          "s3:GetObject",          "s3:PutObject",          "s3:ListBucket"        ]        Resource = [          aws_s3_bucket.raw_data.arn,          "${aws_s3_bucket.raw_data.arn}/*"        ]      }    ]  })}
# Instance profile for EC2resource "aws_iam_instance_profile" "data_processor_profile" {  name = "data-processor-profile"  role = aws_iam_role.data_processor_role.name}

Multi-Environment Setup with Variables

hcl

# terraform/variables.tfvariable "environment" {  description = "Environment name (dev, staging, prod)"  type        = string  default     = "dev"}
variable "bucket_prefix" {  description = "Prefix for S3 bucket names"  type        = string}
variable "instance_type" {  description = "EC2 instance type"  type        = string  default     = "t3.medium"}
variable "aws_region" {  description = "AWS region"  type        = string  default     = "us-east-1"}
# terraform/main.tfresource "aws_s3_bucket" "data_bucket" {  bucket = "${var.bucket_prefix}-${var.environment}-data"    tags = {    Environment = var.environment    ManagedBy   = "terraform"  }}

# terraform/variables.tfvariable "environment" {  description = "Environment name (dev, staging, prod)"  type        = string  default     = "dev"}
variable "bucket_prefix" {  description = "Prefix for S3 bucket names"  type        = string}
variable "instance_type" {  description = "EC2 instance type"  type        = string  default     = "t3.medium"}
variable "aws_region" {  description = "AWS region"  type        = string  default     = "us-east-1"}
# terraform/main.tfresource "aws_s3_bucket" "data_bucket" {  bucket = "${var.bucket_prefix}-${var.environment}-data"    tags = {    Environment = var.environment    ManagedBy   = "terraform"  }}

Apply with variables:

bash

terraform -chdir=terraform apply \  -var="environment=prod" \  -var="bucket_prefix=mycompany" \  -var="instance_type=t3.large"

terraform -chdir=terraform apply \  -var="environment=prod" \  -var="bucket_prefix=mycompany" \  -var="instance_type=t3.large"

Output Values for Integration

hcl

# terraform/outputs.tfoutput "s3_bucket_name" {  description = "Name of the S3 bucket"  value       = aws_s3_bucket.raw_data.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.raw_data.arn}
output "ec2_instance_id" {  description = "ID of the EC2 instance"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of the EC2 instance"  value       = aws_instance.data_processor.public_ip}
# View outputs# terraform -chdir=terraform output

# terraform/outputs.tfoutput "s3_bucket_name" {  description = "Name of the S3 bucket"  value       = aws_s3_bucket.raw_data.id}
output "s3_bucket_arn" {  description = "ARN of the S3 bucket"  value       = aws_s3_bucket.raw_data.arn}
output "ec2_instance_id" {  description = "ID of the EC2 instance"  value       = aws_instance.data_processor.id}
output "ec2_public_ip" {  description = "Public IP of the EC2 instance"  value       = aws_instance.data_processor.public_ip}
# View outputs# terraform -chdir=terraform output

Common Workflows

Initial Setup

bash

# Clone the repositorygit clone https://github.com/josephmachado/iac-for-data-engineering-terraform-.gitcd iac-for-data-engineering-terraform-
# Update bucket name in terraform/main.tf to be globally unique# Edit terraform/main.tf and change bucket name
# Initialize and applyterraform -chdir=terraform initterraform -chdir=terraform validateterraform -chdir=terraform fmtterraform -chdir=terraform apply

# Clone the repositorygit clone https://github.com/josephmachado/iac-for-data-engineering-terraform-.gitcd iac-for-data-engineering-terraform-
# Update bucket name in terraform/main.tf to be globally unique# Edit terraform/main.tf and change bucket name
# Initialize and applyterraform -chdir=terraform initterraform -chdir=terraform validateterraform -chdir=terraform fmtterraform -chdir=terraform apply

Verify Resources Created

bash

# List S3 bucketsaws s3 ls
# Check EC2 instancesaws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress}' \  --output table
# View Terraform stateterraform -chdir=terraform state listcat terraform/terraform.tfstate | jq -r '.resources[] | [.type, .name] | join(",")'

# List S3 bucketsaws s3 ls
# Check EC2 instancesaws ec2 describe-instances \  --filters "Name=instance-state-name,Values=running" \  --query 'Reservations[].Instances[].{ID:InstanceId, Name:Tags[?Key==`Name`].Value, Type:InstanceType, State:State.Name, PublicIP:PublicIpAddress}' \  --output table
# View Terraform stateterraform -chdir=terraform state listcat terraform/terraform.tfstate | jq -r '.resources[] | [.type, .name] | join(",")'

Update Infrastructure

bash

# Edit terraform files# Then preview changesterraform -chdir=terraform plan
# Apply changesterraform -chdir=terraform apply

# Edit terraform files# Then preview changesterraform -chdir=terraform plan
# Apply changesterraform -chdir=terraform apply

Clean Up

bash

# Destroy all resourcesterraform -chdir=terraform destroy
# Verify cleanupaws s3 lsaws ec2 describe-instances --filters "Name=instance-state-name,Values=running"

# Destroy all resourcesterraform -chdir=terraform destroy
# Verify cleanupaws s3 lsaws ec2 describe-instances --filters "Name=instance-state-name,Values=running"

Advanced Patterns

Data Lake Structure with Multiple Buckets

hcl

# Raw data bucketresource "aws_s3_bucket" "raw" {  bucket = "${var.bucket_prefix}-raw-${var.environment}"  tags = {    Layer = "raw"  }}
# Processed data bucketresource "aws_s3_bucket" "processed" {  bucket = "${var.bucket_prefix}-processed-${var.environment}"  tags = {    Layer = "processed"  }}
# Curated data bucketresource "aws_s3_bucket" "curated" {  bucket = "${var.bucket_prefix}-curated-${var.environment}"  tags = {    Layer = "curated"  }}
# Lifecycle policy for raw dataresource "aws_s3_bucket_lifecycle_configuration" "raw_lifecycle" {  bucket = aws_s3_bucket.raw.id
  rule {    id     = "archive-old-data"    status = "Enabled"
    transition {      days          = 90      storage_class = "GLACIER"    }
    expiration {      days = 365    }  }}

# Raw data bucketresource "aws_s3_bucket" "raw" {  bucket = "${var.bucket_prefix}-raw-${var.environment}"  tags = {    Layer = "raw"  }}
# Processed data bucketresource "aws_s3_bucket" "processed" {  bucket = "${var.bucket_prefix}-processed-${var.environment}"  tags = {    Layer = "processed"  }}
# Curated data bucketresource "aws_s3_bucket" "curated" {  bucket = "${var.bucket_prefix}-curated-${var.environment}"  tags = {    Layer = "curated"  }}
# Lifecycle policy for raw dataresource "aws_s3_bucket_lifecycle_configuration" "raw_lifecycle" {  bucket = aws_s3_bucket.raw.id
  rule {    id     = "archive-old-data"    status = "Enabled"
    transition {      days          = 90      storage_class = "GLACIER"    }
    expiration {      days = 365    }  }}

Remote State Management

hcl

# Create S3 bucket for stateresource "aws_s3_bucket" "terraform_state" {  bucket = "my-terraform-state-bucket-12345"    lifecycle {    prevent_destroy = true  }}
resource "aws_s3_bucket_versioning" "terraform_state_versioning" {  bucket = aws_s3_bucket.terraform_state.id    versioning_configuration {    status = "Enabled"  }}
# Configure backend (in a separate backend.tf file)# terraform {#   backend "s3" {#     bucket = "my-terraform-state-bucket-12345"#     key    = "data-engineering/terraform.tfstate"#     region = "us-east-1"#   }# }

# Create S3 bucket for stateresource "aws_s3_bucket" "terraform_state" {  bucket = "my-terraform-state-bucket-12345"    lifecycle {    prevent_destroy = true  }}
resource "aws_s3_bucket_versioning" "terraform_state_versioning" {  bucket = aws_s3_bucket.terraform_state.id    versioning_configuration {    status = "Enabled"  }}
# Configure backend (in a separate backend.tf file)# terraform {#   backend "s3" {#     bucket = "my-terraform-state-bucket-12345"#     key    = "data-engineering/terraform.tfstate"#     region = "us-east-1"#   }# }

Troubleshooting

Bucket Name Already Exists

Error: BucketAlreadyExists: The requested bucket name is not available

Solution: S3 bucket names must be globally unique. Change the bucket name in main.tf:

hcl

resource "aws_s3_bucket" "data_bucket" {  bucket = "your-unique-prefix-data-bucket-${random_id.bucket_suffix.hex}"}
resource "random_id" "bucket_suffix" {  byte_length = 4}

resource "aws_s3_bucket" "data_bucket" {  bucket = "your-unique-prefix-data-bucket-${random_id.bucket_suffix.hex}"}
resource "random_id" "bucket_suffix" {  byte_length = 4}

Insufficient IAM Permissions

Error: UnauthorizedOperation or AccessDenied

Solution: Verify IAM user has required permissions:

bash

# Check current user identityaws sts get-caller-identity
# Verify policies attached to useraws iam list-attached-user-policies --user-name your-terraform-user

# Check current user identityaws sts get-caller-identity
# Verify policies attached to useraws iam list-attached-user-policies --user-name your-terraform-user

State Lock Issues

Error: Error acquiring the state lock

Solution:

bash

# Force unlock (use with caution)terraform -chdir=terraform force-unlock LOCK_ID
# Or remove local state lock filerm terraform/.terraform.tfstate.lock.info

# Force unlock (use with caution)terraform -chdir=terraform force-unlock LOCK_ID
# Or remove local state lock filerm terraform/.terraform.tfstate.lock.info

Resource Already Exists

Error: Resource already exists but not in state

Solution: Import existing resource:

bash

# Import S3 bucketterraform -chdir=terraform import aws_s3_bucket.data_bucket my-existing-bucket-name
# Import EC2 instanceterraform -chdir=terraform import aws_instance.data_processor i-1234567890abcdef0

# Import S3 bucketterraform -chdir=terraform import aws_s3_bucket.data_bucket my-existing-bucket-name
# Import EC2 instanceterraform -chdir=terraform import aws_instance.data_processor i-1234567890abcdef0

Terraform State Drift

Error: Resources differ from state

Solution:

bash

# Refresh state to match real infrastructureterraform -chdir=terraform refresh
# Or during plan/applyterraform -chdir=terraform apply -refresh=true

# Refresh state to match real infrastructureterraform -chdir=terraform refresh
# Or during plan/applyterraform -chdir=terraform apply -refresh=true

Region-Specific AMI Issues

Error: Invalid AMI ID for region

Solution: Use data source to find correct AMI:

hcl

data "aws_ami" "amazon_linux_2" {  most_recent = true  owners      = ["amazon"]
  filter {    name   = "name"    values = ["amzn2-ami-hvm-*-x86_64-gp2"]  }}
resource "aws_instance" "data_processor" {  ami           = data.aws_ami.amazon_linux_2.id  instance_type = "t3.medium"}

data "aws_ami" "amazon_linux_2" {  most_recent = true  owners      = ["amazon"]
  filter {    name   = "name"    values = ["amzn2-ami-hvm-*-x86_64-gp2"]  }}
resource "aws_instance" "data_processor" {  ami           = data.aws_ami.amazon_linux_2.id  instance_type = "t3.medium"}

Best Practices

Use Remote State: Store Terraform state in S3 with versioning enabled
Separate Environments: Use workspaces or separate state files for dev/staging/prod
Least Privilege IAM: Use specific IAM policies instead of full access in production
Tag Everything: Add consistent tags for cost tracking and resource management
Version Control: Commit .tf files but exclude terraform.tfstate and .terraform/
Plan Before Apply: Always run terraform plan before apply
Use Variables: Parameterize configurations for reusability
Enable Encryption: Use S3 bucket encryption and EBS encryption for EC2
Implement Lifecycle Policies: Archive or delete old data automatically
Document Dependencies: Use comments to explain resource relationships

Environment Variables

bash

# AWS credentials (preferred over hardcoding)export AWS_ACCESS_KEY_ID=your_access_keyexport AWS_SECRET_ACCESS_KEY=your_secret_keyexport AWS_DEFAULT_REGION=us-east-1
# Terraform variablesexport TF_VAR_environment=devexport TF_VAR_bucket_prefix=mycompanyexport TF_VAR_instance_type=t3.medium

# AWS credentials (preferred over hardcoding)export AWS_ACCESS_KEY_ID=your_access_keyexport AWS_SECRET_ACCESS_KEY=your_secret_keyexport AWS_DEFAULT_REGION=us-east-1
# Terraform variablesexport TF_VAR_environment=devexport TF_VAR_bucket_prefix=mycompanyexport TF_VAR_instance_type=t3.medium

Source and attribution

Source:reason-machines/data-skillsinskills/terraform-data-engineering-infrastructureat commit8e7bd5f

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Terraform Data Engineering Infrastructure

Terraform Data Engineering Infrastructure

What This Project Does

Prerequisites

Installing Prerequisites

Setting Up IAM Permissions

Project Structure

Key Terraform Commands

Initialize Terraform

Plan and Apply Infrastructure

Inspect Infrastructure

Destroy Infrastructure

Configuration Patterns

Basic S3 Bucket for Data Lake

EC2 Instance for Data Processing

IAM Role for EC2 to Access S3

Multi-Environment Setup with Variables

Output Values for Integration

Common Workflows

Initial Setup

Verify Resources Created

Update Infrastructure

Clean Up

Advanced Patterns

Data Lake Structure with Multiple Buckets

Remote State Management

Troubleshooting

Bucket Name Already Exists

Insufficient IAM Permissions

State Lock Issues

Resource Already Exists

Terraform State Drift

Region-Specific AMI Issues

Best Practices

Environment Variables

Source and attribution

More from reason-machines/data-skills

Youtube Channel Analytics Dashboard

Terraform Iac Data Engineering

Retail Etl Pipeline Medallion

Power Bi Salespulse 360 Dashboard

Logifleet Pulse Supply Chain Analytics

Infrastructure Cicd Data Engineering

More in DevOps & Cloud

Playwright Devops

M5 Onboard

Yeet

Runbook

Incident Response

Deploy Checklist