Skip to main content

​
Overview

The frontend is a Next.js application that provides a web interface for the llms.txt Generator. It connects to the backend API via WebSocket for real-time crawl progress.

​
Frontend Features

Real-time Crawling

WebSocket connection shows live progress as pages are crawled

Download llms.txt

Instantly download generated files or get public URLs

Responsive Design

Works on desktop, tablet, and mobile devices

Error Handling

Graceful error messages and retry logic

​
Deployment Options

The frontend can be deployed to several platforms: This guide focuses on Vercel (recommended).

​
Vercel Deployment

​
Install Vercel CLI

npm install -g vercel
Verify installation: 
vercel --version

​
Login to Vercel

vercel login
Choose your authentication method:
  • GitHub (recommended for auto-deployments)
  • GitLab
  • Bitbucket
  • Email

​
Prepare Environment Variables

Before deploying, collect these values:
1

Get Backend URL

Retrieve the Application Load Balancer URL:
cd terraform
terraform output alb_dns_name
Example: llmstxt-alb-1234567890.us-east-1.elb.amazonaws.com
2

Get API Key

From your terraform.tfvars file:
grep api_key terraform/terraform.tfvars
3

Format WebSocket URL

Construct the WebSocket URL:
ws://[ALB_DNS_NAME]/ws/crawl
Example: ws://llmstxt-alb-1234567890.us-east-1.elb.amazonaws.com/ws/crawl
Use wss:// (secure WebSocket) if you configure HTTPS with a custom domain.

​
Deploy Frontend

Navigate to frontend directory and deploy: 
cd frontend
vercel
1

Project Setup

Vercel CLI will prompt for configuration:
? Set up and deploy "~/frontend"? [Y/n] y
? Which scope? Your Account
? Link to existing project? [y/N] n
? What's your project's name? llmstxt-generator
? In which directory is your code located? ./
2

Framework Detection

Vercel auto-detects Next.js:
Auto-detected Project Settings (Next.js):
- Build Command: next build
- Output Directory: .next
- Development Command: next dev --port $PORT

? Want to override the settings? [y/N] n
3

Environment Variables

After initial deployment, add environment variables:
vercel env add NEXT_PUBLIC_WS_URL
# Paste: ws://your-alb-url.amazonaws.com/ws/crawl
# Select: Production, Preview, Development

vercel env add NEXT_PUBLIC_API_KEY
# Paste: your-api-key-from-terraform
# Select: Production, Preview, Development
4

Production Deployment

Deploy to production with environment variables:
vercel --prod

​
Deployment Output

Successful deployment shows: 
🔍  Inspect: https://vercel.com/your-account/llmstxt-generator/...
✅  Production: https://llmstxt-generator.vercel.app [2s]
Frontend is now live! Visit the URL to test.

​
Configure Environment Variables (Alternative Method)

​
Via Vercel Dashboard

1

Open Project Settings

  1. Go to vercel.com/dashboard
  2. Select your project: llmstxt-generator
  3. Navigate to Settings → Environment Variables
2

Add Variables

Add each variable:
KeyValueEnvironments
NEXT_PUBLIC_WS_URLws://your-alb.amazonaws.com/ws/crawlProduction, Preview, Development
NEXT_PUBLIC_API_KEYYour API keyProduction, Preview, Development
3

Redeploy

Trigger a new deployment to apply variables:
vercel --prod
Environment variables prefixed with NEXT_PUBLIC_ are exposed to the browser. Never put secrets without this prefix in client-side code.

​
Test Frontend

​
Access the Application

Visit your Vercel URL: https://llmstxt-generator.vercel.app

​
Verify WebSocket Connection

1

Open Browser DevTools

Press F12 or right-click → Inspect → Console tab
2

Enter URL and Start Crawl

In the web interface:
  1. Enter a URL (e.g., https://example.com)
  2. Set max pages (e.g., 10)
  3. Click “Generate llms.txt”
3

Watch Console Logs

You should see:
  • WebSocket connection established
  • Crawl progress messages
  • Page discoveries
  • Completion message
If you see real-time progress, the frontend is correctly connected to the backend!

​
Configure Custom Domain (Optional)

1

Add Domain in Vercel

  1. Go to Project Settings → Domains
  2. Click “Add”
  3. Enter your domain: llms.yoursite.com
  4. Click “Add”
2

Update DNS Records

Vercel provides DNS records to add:For root domain (yoursite.com):
  • Type: A
  • Name: @
  • Value: 76.76.21.21
For subdomain (llms.yoursite.com):
  • Type: CNAME
  • Name: llms
  • Value: cname.vercel-dns.com
3

Wait for Verification

Vercel automatically provisions SSL certificate (1-5 minutes).
4

Update WebSocket URL

If using HTTPS frontend with HTTP backend:
  • Use wss:// for WebSocket URL (requires HTTPS backend)
  • Or configure backend custom domain with SSL

​
AWS Amplify Deployment (Alternative)

If you prefer to deploy the frontend on AWS:
Terraform already created an Amplify app. Complete the setup:

​
Connect GitHub Repository

1

Get Amplify App ID

cd terraform
terraform output amplify_app_id
2

Open Amplify Console

  1. Go to AWS Amplify Console
  2. Select your app: llmstxt-frontend
3

Connect Repository

  1. Click “Connect branch”
  2. Choose GitHub
  3. Authorize AWS Amplify
  4. Select repository and branch (main or master)
4

Configure Build Settings

Amplify auto-detects Next.js. Verify:
version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
5

Add Environment Variables

In Amplify Console → Environment variables:
  • NEXT_PUBLIC_WS_URL
  • NEXT_PUBLIC_API_KEY
6

Deploy

Save and deploy. Amplify will:
  1. Clone repository
  2. Install dependencies
  3. Build Next.js app
  4. Deploy to CDN

​
Get Amplify URL

terraform output amplify_app_url
Example: https://infra.d1a2b3c4d5e6f7.amplifyapp.com

​
Continuous Deployment

​
Vercel GitHub Integration

Enable automatic deployments on Git push:
1

Connect Git Repository

In Vercel dashboard:
  1. Go to Project Settings → Git
  2. Click “Connect Git Repository”
  3. Select GitHub/GitLab/Bitbucket
  4. Choose repository
2

Configure Branch Deployments

  • Production branch: main or master
  • Preview branches: All other branches
3

Push Changes

Every push triggers automatic deployment:
git add .
git commit -m "Update frontend"
git push origin main
Vercel will:
  • Build and deploy on every push
  • Create preview URLs for PRs
  • Run build checks before merging

​
Frontend Configuration

​
Update Backend URL

If your backend URL changes (e.g., custom domain): 
vercel env rm NEXT_PUBLIC_WS_URL production
vercel env add NEXT_PUBLIC_WS_URL production
# Enter new URL

vercel --prod  # Redeploy

​
Configure CORS

Ensure backend allows your frontend domain: Edit terraform/terraform.tfvars: 
cors_origins = "https://llmstxt-generator.vercel.app,https://yourdomain.com"
Apply changes: 
cd terraform
terraform apply
Force ECS redeploy: 
aws ecs update-service \
  --cluster llmstxt-cluster \
  --service llmstxt-api-service \
  --force-new-deployment

​
Troubleshooting

Check browser console for errors:
  • CORS error: Add frontend domain to cors_origins in backend
  • Connection refused: Verify backend ALB is accessible
  • Invalid API key: Check NEXT_PUBLIC_API_KEY matches backend
Test backend WebSocket manually:
websocat "ws://your-alb.amazonaws.com/ws/crawl?api_key=YOUR_KEY"
Common issues:
  1. Missing dependencies: Check package.json
  2. Build errors: Run locally first: 
    cd frontend
npm install
npm run build
  3. Environment variables: Ensure they’re set in Vercel
Verify Next.js routing:
  • Check frontend/pages/ or frontend/app/ directory structure
  • Ensure index.js or page.js exists
  • Review Vercel build logs for errors
When frontend uses HTTPS but backend uses HTTP:Option 1: Use HTTP WebSocket on HTTPS page (browser blocks this)Option 2: Configure HTTPS for backend:
  1. Set up custom domain for ALB
  2. Use ACM certificate (already created by Terraform)
  3. Update WebSocket URL to wss://

​
Performance Optimization

​
Enable Vercel Analytics

cd frontend
npm install @vercel/analytics
Add to pages/_app.js: 
import { Analytics } from '@vercel/analytics/react';

export default function App({ Component, pageProps }) {
  return (
    <>
      <Component {...pageProps} />
      <Analytics />
    </>
  );
}

​
Configure Caching

Optimize static assets in next.config.js: 
module.exports = {
  async headers() {
    return [
      {
        source: '/static/:path*',
        headers: [
          {
            key: 'Cache-Control',
            value: 'public, max-age=31536000, immutable',
          },
        ],
      },
    ];
  },
};

​
Next Steps

Monitoring & Logs

Set up CloudWatch monitoring and alerts for your deployment

Build docs developers (and LLMs) love

Get started for freeTalk to us