Skip to main content

​
Overview

Terraform is used to provision all AWS infrastructure including ECS, ECR, Application Load Balancer, Lambda, and monitoring resources.

​
What Terraform Creates

Compute

  • ECS Fargate cluster and service
  • ECR repository for Docker images
  • Lambda function for scheduled crawls

Networking

  • Application Load Balancer
  • Target groups and listeners
  • Security groups

Monitoring

  • CloudWatch log groups
  • CloudWatch alarms (10 alerts)
  • SNS topic for notifications

Automation

  • EventBridge cron schedule
  • Lambda permissions
  • IAM roles and policies

​
Configure Terraform Variables

​
Clone or Navigate to Terraform Directory

cd /path/to/llmstxt-generator
cd terraform

​
Copy Example Variables File

cp terraform.tfvars.example terraform.tfvars

​
Edit Configuration

Open terraform.tfvars in your editor: 
nano terraform.tfvars
# or
vim terraform.tfvars
# or
code terraform.tfvars

​
Required Variables

Fill in all the following variables with values collected from previous steps: 
# AWS Configuration
aws_region  = "us-east-1"
environment = "production"

​
Generate Security Keys

Generate strong random keys for API authentication: 
# Generate API key for WebSocket authentication
openssl rand -base64 32
# Example output: 7X9k2L5m8P1q4R6s...

# Generate cron secret for Lambda authentication
openssl rand -base64 32
# Example output: 3A6b9C2d5E8f1G4h...
Add these to your terraform.tfvars: 
api_key     = "7X9k2L5m8P1q4R6s..."  # From above
cron_secret = "3A6b9C2d5E8f1G4h..."  # From above
Keep these secrets secure! They protect your API from unauthorized access.

​
Build Lambda Deployment Package

The Lambda function requires dependencies packaged into a ZIP file.
1

Navigate to Backend Directory

cd ../backend
2

Run Build Script

chmod +x deployment/build_lambda.sh
./deployment/build_lambda.sh
Expected output:
Building Lambda deployment package...
Installing dependencies...
Copying application code...
Creating deployment package...
Lambda deployment package created: deployment/lambda-deployment.zip
Size: 68M
3

Copy Package to Root

Terraform expects the package in backend/ directory:
cp deployment/lambda-deployment.zip ./lambda-deployment.zip
4

Verify Package

ls -lh lambda-deployment.zip
# Expected: ~60-70MB file
The Lambda package includes all Python dependencies (requests, boto3, etc.) needed to trigger the recrawl endpoint.

​
Initialize Terraform

Prepare Terraform to deploy infrastructure. 
cd ../terraform
terraform init
Expected output: 
Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.x.x...

Terraform has been successfully initialized!
If initialization succeeds, you’re ready to deploy!

​
Review Infrastructure Plan

Before deploying, review what Terraform will create: 
terraform plan
This shows a detailed list of resources to be created: 
Terraform will perform the following actions:

  # aws_cloudwatch_event_rule.recrawl_schedule will be created
  # aws_cloudwatch_event_target.lambda_target will be created
  # aws_cloudwatch_log_group.ecs_logs will be created
  # aws_cloudwatch_log_group.lambda_logs will be created
  # aws_cloudwatch_metric_alarm.ecs_no_running_tasks will be created
  # ... (10 total alarms)
  # aws_ecr_repository.llmstxt_api will be created
  # aws_ecs_cluster.llmstxt will be created
  # aws_ecs_service.llmstxt_api will be created
  # aws_ecs_task_definition.llmstxt_api will be created
  # aws_iam_role.ecs_execution_role will be created
  # aws_iam_role.ecs_task_role will be created
  # aws_iam_role.lambda_execution will be created
  # aws_lambda_function.llmstxt_auto_update will be created
  # aws_lambda_permission.allow_eventbridge will be created
  # aws_lb.llmstxt will be created
  # aws_lb_listener.http will be created
  # aws_lb_listener.https will be created
  # aws_lb_target_group.llmstxt_api will be created
  # aws_s3_bucket.lambda_deployments will be created
  # aws_s3_object.lambda_package will be created
  # aws_security_group.alb will be created
  # aws_security_group.ecs_tasks will be created
  # aws_sns_topic.llmstxt_alerts will be created
  # aws_sns_topic_subscription.email_alerts will be created

Plan: 30+ to add, 0 to change, 0 to destroy.
Review the plan carefully. Ensure VPC and subnet IDs are correct before proceeding.

​
Deploy Infrastructure

Apply the Terraform configuration to create all AWS resources: 
terraform apply
Terraform will show the plan again and prompt for confirmation: 
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:
Type yes and press Enter.

​
Deployment Progress

Terraform will create resources in order (respecting dependencies):
1

IAM Roles & Policies (1 min)

Creates execution roles for ECS and Lambda with necessary permissions.
2

Network Resources (2 min)

Creates security groups, Application Load Balancer, target groups, and listeners.
3

Compute Resources (2 min)

Creates ECR repository, ECS cluster, task definition, and service.
4

Lambda & Automation (1 min)

Uploads Lambda package to S3, creates Lambda function and EventBridge schedule.
5

Monitoring & Alarms (2 min)

Creates CloudWatch log groups, SNS topic, and 10 metric alarms.
Total deployment time: 5-10 minutes

​
Deployment Complete

When finished, Terraform outputs important values: 
Apply complete! Resources: 30 added, 0 changed, 0 destroyed.

Outputs:

alb_dns_name = "llmstxt-alb-1234567890.us-east-1.elb.amazonaws.com"
alb_url = "http://llmstxt-alb-1234567890.us-east-1.elb.amazonaws.com"
ecr_repository_url = "123456789012.dkr.ecr.us-east-1.amazonaws.com/llmstxt-api"
ecs_cluster_name = "llmstxt-cluster"
ecs_service_name = "llmstxt-api-service"
lambda_function_name = "llmstxt-auto-update"
Infrastructure successfully deployed! Save these output values.

​
Retrieve Terraform Outputs

You can retrieve outputs anytime: 
# All outputs
terraform output

# Specific output
terraform output ecr_repository_url
terraform output alb_dns_name

​
Verify Resource Creation

​
Check ECS Cluster

aws ecs describe-clusters \
  --clusters llmstxt-cluster \
  --region us-east-1
Expected status: ACTIVE

​
Check ECR Repository

aws ecr describe-repositories \
  --repository-names llmstxt-api \
  --region us-east-1
Repository should exist but have no images yet (that’s next step).

​
Check Load Balancer

aws elbv2 describe-load-balancers \
  --names llmstxt-alb \
  --region us-east-1
Expected state: active

​
Confirm SNS Subscription

Check your email for SNS subscription confirmation:
  1. Look for email from AWS Notifications <[email protected]>
  2. Subject: “AWS Notification - Subscription Confirmation”
  3. Click “Confirm subscription” link
You won’t receive CloudWatch alerts until you confirm the SNS subscription!

​
Terraform State Management

​
Local State File

Terraform stores infrastructure state in terraform.tfstate. This file is critical for managing resources.
Protect terraform.tfstate: Never commit to version control! Add to .gitignore.
For production or team environments, store state remotely:
  1. Create S3 bucket for state:
aws s3 mb s3://llmstxt-terraform-state-YOUR-ACCOUNT-ID
  1. Enable versioning:
aws s3api put-bucket-versioning \
  --bucket llmstxt-terraform-state-YOUR-ACCOUNT-ID \
  --versioning-configuration Status=Enabled
  1. Add backend configuration to main.tf:
terraform {
  backend "s3" {
    bucket = "llmstxt-terraform-state-YOUR-ACCOUNT-ID"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
  }
}
  1. Migrate state:
terraform init -migrate-state

​
Common Issues & Solutions

Problem: Invalid subnet IDs in terraform.tfvarsSolution: Verify subnet IDs:
aws ec2 describe-subnets \
  --subnet-ids subnet-xxxxx subnet-yyyyy
Ensure subnets exist and are in different availability zones.
Problem: lambda-deployment.zip missingSolution: Build Lambda package:
cd ../backend
./deployment/build_lambda.sh
cp deployment/lambda-deployment.zip ./
cd ../terraform
Problem: ACM certificate waiting for DNS validationSolution: This is expected. Certificate validation happens after DNS records are added. The ALB will use HTTP (port 80) until then.
Problem: AWS user lacks required permissionsSolution: Attach AdministratorAccess policy (or create custom policy with required permissions).

​
Update Infrastructure

To modify infrastructure after initial deployment:
  1. Edit terraform.tfvars or *.tf files
  2. Review changes: terraform plan
  3. Apply changes: terraform apply
Terraform only modifies changed resources.

​
Destroy Infrastructure

Destructive Action: This deletes ALL AWS resources created by Terraform.
To tear down the entire infrastructure: 
terraform destroy
Type yes to confirm deletion.

​
Next Steps

Docker & ECS Deployment

Build Docker image and deploy to ECS Fargate

Build docs developers (and LLMs) love

Get started for freeTalk to us