AWS Infrastructure

Infrastructure Overview

The llms.txt Generator infrastructure is fully defined as code using Terraform and deployed on AWS. The architecture is designed for high availability, scalability, and cost-effectiveness.

Infrastructure Diagram

┌─────────────────────────────────────────────────────────┐
│                     AWS Cloud                            │
│                                                          │
│  ┌────────────────────────────────────────────────┐    │
│  │            Application Load Balancer            │    │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐     │    │
│  │  │ HTTP:80  │  │HTTPS:443 │  │ WSS      │     │    │
│  │  └─────┬────┘  └────┬─────┘  └────┬─────┘     │    │
│  └────────┼────────────┼─────────────┼───────────┘    │
│           │            │             │                  │
│           └────────────┴─────────────┘                  │
│                        │                                │
│           ┌────────────▼────────────┐                  │
│           │   Target Group (IP)      │                  │
│           │   Health: /health        │                  │
│           └────────────┬─────────────┘                  │
│                        │                                │
│           ┌────────────▼────────────┐                  │
│           │      ECS Cluster         │                  │
│           │  ┌─────────────────┐    │                  │
│           │  │  Fargate Task   │    │                  │
│           │  │  ┌───────────┐  │    │                  │
│           │  │  │ Container │  │    │                  │
│           │  │  │ FastAPI   │  │    │                  │
│           │  │  │ Port:8000 │  │    │                  │
│           │  │  └───────────┘  │    │                  │
│           │  │  CPU: 0.5 vCPU  │    │                  │
│           │  │  RAM: 1 GB      │    │                  │
│           │  └─────────────────┘    │                  │
│           └─────────────────────────┘                  │
│                                                          │
│  ┌───────────────────────────────────────────────┐    │
│  │          AWS Lambda Function                   │    │
│  │  Name: llmstxt-auto-update                    │    │
│  │  Runtime: Python 3.11                         │    │
│  │  Timeout: 10 minutes                          │    │
│  │  Memory: 512 MB                               │    │
│  └────────────▲──────────────────────────────────┘    │
│               │                                         │
│  ┌────────────┴──────────────────────────────────┐    │
│  │         EventBridge Rule                       │    │
│  │  Schedule: cron(0 */6 * * ? *)                │    │
│  │  Trigger: Every 6 hours                        │    │
│  └────────────────────────────────────────────────┘    │
│                                                          │
└──────────────────────────────────────────────────────────┘

         ┌──────────────┐      ┌──────────────┐
         │   Supabase   │      │ Cloudflare R2│
         │  PostgreSQL  │      │    Storage   │
         └──────────────┘      └──────────────┘
              External              External

Core Components

1. ECS Fargate Cluster

Container Orchestration

Purpose: Runs the FastAPI backend application in Docker containersConfiguration:

Launch Type: Fargate (serverless)
Task CPU: 512 units (0.5 vCPU)
Task Memory: 1024 MB (1 GB)
Container Port: 8000
Desired Count: 1 task
Network Mode: awsvpc

Key Features:

Container Insights: Enabled for enhanced monitoring
Auto-scaling: Can scale based on CPU/memory metrics
Health Checks: ALB performs health checks on /health endpoint
Zero Downtime Deployments: Rolling updates when new images are pushed

# terraform/ecs.tf excerpt
resource "aws_ecs_task_definition" "llmstxt_api" {
  family                   = "llmstxt-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "512"
  memory                   = "1024"
  execution_role_arn       = aws_iam_role.ecs_execution_role.arn
  task_role_arn            = aws_iam_role.ecs_task_role.arn
}

2. Application Load Balancer (ALB)

Load Balancing & SSL Termination

Purpose: Distributes HTTP/HTTPS traffic to ECS tasksConfiguration:

Type: Application Load Balancer
Scheme: Internet-facing
Listeners: HTTP (80), HTTPS (443)
Target Type: IP (required for Fargate)
Health Check Path: /health

Listeners:

HTTP Listener (Port 80)

Forwards traffic to target group
Can be configured to redirect to HTTPS
Used for health checks

HTTPS Listener (Port 443)

SSL/TLS termination using ACM certificate
Forwards decrypted traffic to backend
Security Policy: ELBSecurityPolicy-2016-08
Supports WebSocket (WSS) connections

Health Check Configuration:

health_check {
  enabled             = true
  healthy_threshold   = 2
  unhealthy_threshold = 3
  timeout             = 5
  interval            = 30
  path                = "/health"
  matcher             = "200"
}

3. ECR (Elastic Container Registry)

Docker Image Registry

Purpose: Stores and manages Docker images for the FastAPI backendFeatures:

Image vulnerability scanning on push
Mutable image tags (allows :latest updates)
Private repository with IAM-based access
Integrated with ECS for seamless deployments

Deployment Workflow:

# 1. Build Docker image
docker build -t llmstxt-api:latest .

# 2. Authenticate with ECR
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin <ECR_URL>

# 3. Tag image
docker tag llmstxt-api:latest <ECR_URL>/llmstxt-api:latest

# 4. Push to ECR
docker push <ECR_URL>/llmstxt-api:latest

# 5. Force ECS service update
aws ecs update-service --cluster llmstxt-cluster \
  --service llmstxt-api-service --force-new-deployment

4. Lambda Function

Scheduled Recrawl Executor

Purpose: Triggers automated recrawls of enrolled sitesConfiguration:

Runtime: Python 3.11
Timeout: 600 seconds (10 minutes)
Memory: 512 MB
Trigger: EventBridge cron schedule
Handler: lambda_handler.lambda_handler

Environment Variables:

API_URL = "https://llmstxt-backend.yourdomain.com"
CRON_SECRET = "<generated_secret>"

Execution Flow:

EventBridge triggers Lambda every 6 hours
Lambda sends HTTP POST to /internal/cron/recrawl
Backend processes recrawl in background task
Lambda completes quickly, backend handles async work

The Lambda function deployment package is built using backend/deployment/build_lambda.sh and includes all Python dependencies (~70MB).

5. EventBridge Rule

Cron Scheduler

Purpose: Triggers Lambda function on a scheduleSchedule Expression: cron(0 */6 * * ? *)Frequency: Every 6 hours (00:00, 06:00, 12:00, 18:00 UTC)

resource "aws_cloudwatch_event_rule" "recrawl_schedule" {
  name                = "llmstxt-recrawl-schedule"
  schedule_expression = "cron(0 */6 * * ? *)"
}

6. Security Groups

ALB Security Group

Ingress:

Port 80 from 0.0.0.0/0
Port 443 from 0.0.0.0/0

Egress:

All traffic to 0.0.0.0/0

ECS Tasks Security Group

Ingress:

Port 8000 from ALB security group only

Egress:

All traffic to 0.0.0.0/0 (for external API calls)

7. IAM Roles

ECS Execution Role

Purpose: Allows ECS to pull images from ECR and write logs to CloudWatchManaged Policies:

AmazonECSTaskExecutionRolePolicy

Permissions:

Pull images from ECR
Create and write CloudWatch log streams

ECS Task Role

Purpose: Grants permissions to the running containerCustom Permissions:

CloudWatch Logs write access
Can be extended for S3, Secrets Manager, etc.

Lambda Execution Role

Purpose: Allows Lambda to execute and write logsManaged Policies:

AWSLambdaBasicExecutionRole

Custom Permissions:

CloudWatch Logs write access

8. CloudWatch Log Groups

ECS Logs

Log Group: /ecs/llmstxt-apiRetention: 14 daysContents: Application logs, errors, crawl activity

Lambda Logs

Log Group: /aws/lambda/llmstxt-auto-updateRetention: 14 daysContents: Scheduled recrawl execution logs

Viewing Logs:

# ECS application logs
aws logs tail /ecs/llmstxt-api --follow

# Lambda function logs
aws logs tail /aws/lambda/llmstxt-auto-update --follow

9. S3 Bucket (Lambda Deployments)

Lambda Package Storage

Purpose: Stores Lambda deployment package (.zip)Bucket Name: llmstxt-lambda-deployments-<account-id>Objects:

lambda-deployment.zip (~70MB with dependencies)

Network Architecture

VPC Configuration

The infrastructure uses your default VPC or a specified VPC with at least 2 subnets in different availability zones for high availability.

Requirements:

VPC: Must have internet gateway for public access
Subnets: Minimum 2 subnets in different AZs
Route Tables: Subnets must route to internet gateway
Public IPs: ECS tasks require public IPs for external API calls

Terraform Variables:

variable "vpc_id" {
  description = "VPC ID for ECS and ALB"
  type        = string
}

variable "subnet_ids" {
  description = "Subnet IDs (must be in different AZs)"
  type        = list(string)
}

Terraform State Management

Terraform state contains sensitive information. Use remote state storage (S3 + DynamoDB) for production deployments.

Recommended Backend Configuration:

terraform {
  backend "s3" {
    bucket         = "your-terraform-state-bucket"
    key            = "llmstxt/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-lock-table"
  }
}

Cost Optimization

ECS Fargate

Current: 0.5 vCPU, 1 GB RAMCost: ~$12-15/month (1 task running 24/7)Optimization: Scale to zero during off-hours if possible

Lambda

Invocations: 4 per day (every 6 hours)Cost: Less than $1/month (free tier covers this)Optimization: Already optimized with fast execution

CloudWatch Logs

Retention: 14 daysCost: ~$1-3/month (depends on log volume)Optimization: Reduce retention or export to S3

ALB

Cost: ~$16-20/month (base + LCU charges)Optimization: Consider CloudFront for caching

Total Estimated Cost: $30-40/month for AWS infrastructure

Monitoring & Alerts

CloudWatch Alarms

# Example: High CPU alarm
resource "aws_cloudwatch_metric_alarm" "ecs_cpu_high" {
  alarm_name          = "llmstxt-ecs-cpu-high"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "CPUUtilization"
  namespace           = "AWS/ECS"
  period              = "300"
  statistic           = "Average"
  threshold           = "80"
  alarm_description   = "ECS CPU utilization is too high"
}

Key Metrics to Monitor

ECS CPU Utilization: Should stay below 80%
ECS Memory Utilization: Should stay below 80%
ALB Target Response Time: Should be < 1 second
ALB Healthy Host Count: Should match desired count
Lambda Errors: Should be zero
Lambda Duration: Should be < 60 seconds

System Design

Infrastructure Overview

Infrastructure Diagram

Core Components

1. ECS Fargate Cluster

Container Orchestration

2. Application Load Balancer (ALB)

Load Balancing & SSL Termination

3. ECR (Elastic Container Registry)

Docker Image Registry

4. Lambda Function

Scheduled Recrawl Executor

5. EventBridge Rule

Cron Scheduler

6. Security Groups

ALB Security Group

ECS Tasks Security Group

7. IAM Roles

8. CloudWatch Log Groups

ECS Logs

Lambda Logs

9. S3 Bucket (Lambda Deployments)

Lambda Package Storage

Network Architecture

VPC Configuration

Terraform State Management

Cost Optimization

ECS Fargate

Lambda

CloudWatch Logs

ALB

Monitoring & Alerts

CloudWatch Alarms

Key Metrics to Monitor

Next Steps

Data Flow

Deployment Guide

Build docs developers (and LLMs) love

System Design

​Infrastructure Overview

​Infrastructure Diagram

​Core Components

​1. ECS Fargate Cluster

Container Orchestration

​2. Application Load Balancer (ALB)

Load Balancing & SSL Termination

​3. ECR (Elastic Container Registry)

Docker Image Registry

​4. Lambda Function

Scheduled Recrawl Executor

​5. EventBridge Rule

Cron Scheduler

​6. Security Groups

ALB Security Group

ECS Tasks Security Group

​7. IAM Roles

​8. CloudWatch Log Groups

ECS Logs

Lambda Logs

​9. S3 Bucket (Lambda Deployments)

Lambda Package Storage

​Network Architecture

​VPC Configuration

​Terraform State Management

​Cost Optimization

ECS Fargate

Lambda

CloudWatch Logs

ALB

​Monitoring & Alerts

​CloudWatch Alarms

​Key Metrics to Monitor

​Next Steps

Data Flow

Deployment Guide

Build docs developers (and LLMs) love

Infrastructure Overview

Infrastructure Diagram

Core Components

1. ECS Fargate Cluster

2. Application Load Balancer (ALB)

3. ECR (Elastic Container Registry)

4. Lambda Function

5. EventBridge Rule

6. Security Groups

7. IAM Roles

8. CloudWatch Log Groups

9. S3 Bucket (Lambda Deployments)

Network Architecture

VPC Configuration

Terraform State Management

Cost Optimization

Monitoring & Alerts

CloudWatch Alarms

Key Metrics to Monitor

Next Steps