Architecture Overview

​

System Architecture

​

Architecture Diagram

┌─────────────────┐
│   Next.js UI    │  (Vercel)
│   WebSocket     │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  FastAPI API    │  (AWS ECS Fargate)
│  WebSocket      │  ← CloudFlare Tunnel
└────────┬────────┘
         │
    ┌────┴────────────────┐
    │                     │
    ▼                     ▼
┌──────────┐      ┌──────────────┐
│ Crawler  │      │   Storage    │
│ Playwright      │   Supabase   │
│ Brightdata      │   R2 CDN     │
└──────────┘      └──────────────┘
         ▲
         │
    ┌────┴────┐
    │ Lambda  │  (Scheduled Recrawls)
    │ Cron    │  EventBridge: Every 6h
    └─────────┘

​

Core Components

​

Component Interactions

​

User-Initiated Crawl Flow

1. User submits a website URL through the Next.js frontend
2. WebSocket connection established with FastAPI backend
3. Crawler engine performs BFS traversal or sitemap-based crawl
4. Real-time logs streamed back to frontend via WebSocket
5. Generated llms.txt saved to R2 storage
6. Public URL returned to user for immediate access
7. Metadata saved to Supabase for scheduled updates (if enabled)

​

Automated Recrawl Flow

1. EventBridge rule triggers Lambda function every 6 hours
2. Lambda queries Supabase for sites due for recrawl
3. HTTP POST sent to FastAPI /internal/cron/recrawl endpoint
4. Background task processes each site sequentially
5. Updated llms.txt saved to R2, replacing previous version
6. Metadata updated in Supabase with new timestamp

​

Design Principles

​

Scalability

• Containerized deployment on ECS Fargate allows horizontal scaling
• Auto-scaling based on CPU/memory utilization
• Stateless API design enables multiple backend instances
• Queue-based crawling prevents memory overflow on large sites

​

Reliability

• Health checks at ALB and ECS task levels
• Automatic failover for unhealthy containers
• Retry logic for failed page fetches (3x max pages attempted)
• CloudWatch monitoring for logs and metrics

​

Security

• API key authentication for WebSocket endpoints
• JWT token support for time-limited access
• CORS protection with configurable origins
• TLS/SSL encryption for all traffic (HTTPS/WSS)
• Secrets management via environment variables

​

Performance

• Concurrent crawling with asyncio for I/O-bound operations
• CDN delivery of generated llms.txt files via R2
• Sitemap detection for faster discovery of pages
• Brightdata integration for JavaScript-heavy sites

​

Next Steps