Skip to main content

​
Architecture Overview

The llms.txt Generator uses a modern serverless and containerized architecture deployed on AWS, with managed services for database and storage.

​
Key Components

Backend API

FastAPI application running on ECS Fargate behind an Application Load Balancer

Database

Supabase PostgreSQL for tracking crawl sites and metadata

Storage

Cloudflare R2 for storing generated llms.txt files

Frontend

Next.js application deployed on Vercel

​
Architecture Diagram

​
Infrastructure Components

​
AWS Resources

  • ECS Cluster: Container orchestration
  • Task Definition: 0.5 vCPU, 1GB RAM
  • Service: 1 running task with auto-healing
  • ECR Repository: Docker image registry
  • Application Load Balancer: Routes HTTP/HTTPS traffic
  • Target Group: Health checks on /health endpoint
  • Security Groups: ALB (80/443) → ECS Tasks (8000)
  • VPC & Subnets: Requires 2+ subnets in different AZs
  • Lambda Function: Triggers recrawl endpoint
  • EventBridge Rule: Cron schedule (every 6 hours)
  • S3 Bucket: Lambda deployment package storage
  • CloudWatch Logs: Application and Lambda logs (14-day retention)
  • CloudWatch Metrics: CPU, Memory, Response Time
  • CloudWatch Alarms: 10 alarms for critical metrics
  • SNS Topic: Email alerts for incidents
  • IAM Roles: Least-privilege access for ECS and Lambda
  • ACM Certificate: SSL/TLS for HTTPS
  • Secrets: Environment variables in task definitions

​
External Services

Supabase

  • PostgreSQL database
  • Free tier supported
  • Stores site metadata and crawl history

Cloudflare R2

  • S3-compatible object storage
  • Public URL access
  • Stores generated llms.txt files

Vercel

  • Frontend hosting
  • Automatic deployments from Git
  • Edge network CDN

Brightdata (Optional)

  • JavaScript-heavy site scraping
  • Scraping Browser API
  • Pay-per-use pricing

​
Deployment Flow

1

Prerequisites Setup

Install required tools (AWS CLI, Terraform, Docker) and create accounts (AWS, Supabase, Cloudflare)
2

AWS Configuration

Configure AWS CLI and identify VPC/subnet resources for ECS deployment
3

Database & Storage

Set up Supabase database with migrations and configure Cloudflare R2 bucket
4

Terraform Variables

Configure terraform.tfvars with all service credentials and settings
5

Infrastructure Deployment

Run terraform apply to create all AWS resources (ECR, ECS, ALB, Lambda, monitoring)
6

Docker Build & Deploy

Build Docker image, push to ECR, and deploy to ECS
7

Frontend Deployment

Deploy Next.js frontend to Vercel with backend API URL
8

DNS & SSL

Configure custom domain and SSL certificate (optional)

​
Resource Requirements

​
Compute Resources

ComponentSpecificationCost Estimate
ECS Fargate Task0.5 vCPU, 1GB RAM~$15/month
Application Load BalancerStandard ALB~$16/month
Lambda Function512MB, runs every 6 hoursLess than $1/month
CloudWatch Logs14-day retention~$2/month
Total AWS infrastructure costs: approximately $35-40/month for production deployment.

​
External Services

  • Supabase: Free tier (500MB database, 1GB storage)
  • Cloudflare R2: Free tier (10GB storage, 1M Class A operations)
  • Vercel: Free tier (100GB bandwidth)
  • Brightdata: Pay-per-use (optional, ~$0.001 per page)

​
Deployment Time

Expected time to complete full deployment:
  • Prerequisites & Account Setup: 30-60 minutes (one-time)
  • Terraform Infrastructure: 5-10 minutes
  • Docker Build & Push: 5-10 minutes
  • ECS Service Deployment: 3-5 minutes
  • Frontend Deployment: 2-3 minutes
First-time deployments may take longer due to account setup, DNS propagation, and SSL certificate validation.

​
Next Steps

Prerequisites

Install required tools and create service accounts

AWS Setup

Configure AWS CLI and identify network resources

Database & Storage

Set up Supabase and Cloudflare R2

Terraform Deployment

Deploy infrastructure with Terraform

Build docs developers (and LLMs) love

Get started for freeTalk to us