Self-hosting

LLM Gateway can be self-hosted on your own infrastructure, giving you complete control over your data, configuration, and deployment environment. This guide covers deployment options and setup instructions.

Prerequisites

Before deploying LLM Gateway, ensure you have:

Docker (version 20.10 or later) and Docker Compose
PostgreSQL (version 14 or later)
Redis (version 6 or later)
Node.js (version 20 or later) - for building from source
pnpm (version 8 or later) - for package management

For production deployments, we recommend at least 2 CPU cores and 4GB RAM for the gateway service, plus resources for PostgreSQL and Redis.

Quick start with Docker Compose

The fastest way to self-host LLM Gateway is using Docker Compose, which sets up all required services automatically.

Clone the repository

git clone https://github.com/theopenco/llmgateway.git
cd llmgateway

Configure environment variables

Copy the example environment file and configure your settings:

cp .env.example .env

Edit .env and set at least these required variables:

# Database
DATABASE_URL=postgres://postgres:pw@localhost:5432/llmgateway

# Redis
REDIS_URL=redis://localhost:6379

# API URLs
GATEWAY_URL=http://localhost:4001
API_URL=http://localhost:4002
UI_URL=http://localhost:3002

# Authentication (generate secure random strings)
BETTER_AUTH_SECRET=your-secure-secret-here
BETTER_AUTH_URL=http://localhost:4002

# Provider API keys (add the providers you want to use)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...

View all available environment variables

See the environment variables reference for a complete list of configuration options, including provider API keys, feature flags, and performance tuning settings.

Start services with Docker Compose

Start all services (PostgreSQL, Redis, Gateway, API, UI):

docker compose up -d

This will start:

PostgreSQL on localhost:5432
Redis on localhost:6379
Gateway API on localhost:4001
Management API on localhost:4002
UI Dashboard on localhost:3002

Use docker compose logs -f to follow logs from all services.

Initialize the database

Run database migrations and seed initial data:

# Install dependencies
pnpm install

# Run setup (pushes schema and seeds data)
pnpm run setup

This creates the necessary database tables and adds initial configuration.

Verify the deployment

Check that all services are healthy:

# Check gateway health
curl http://localhost:4001/

# Check API health
curl http://localhost:4002/api/health

# Open the UI in your browser
open http://localhost:3002

All health checks should return 200 OK with status information.

Docker Compose configuration

The included docker-compose.yml defines the following services:

docker-compose.yml

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: llmgateway
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: pw
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

volumes:
  postgres_data:
  redis_data:

For production deployments, see the Docker deployment guide for advanced configuration, including SSL, health checks, and resource limits.

Building from source

To build and run LLM Gateway from source:

Install dependencies

pnpm install

Set up the database

Ensure PostgreSQL and Redis are running, then initialize the database:

pnpm run setup

This command:

Pushes the database schema to PostgreSQL
Seeds initial data
Validates the configuration

Build the applications

pnpm build

This builds all services (Gateway, API, UI, Playground, Admin).

Start development servers

pnpm dev

Or start individual services:

# Gateway only
pnpm --filter gateway dev

# API only
pnpm --filter api dev

# UI only
pnpm --filter ui dev

Production deployment

For production environments, consider these deployment options:

Docker deployment

Deploy with Docker using health checks, resource limits, and SSL configuration

Kubernetes deployment

Deploy to Kubernetes with Helm charts, auto-scaling, and high availability

Configuration

Database

LLM Gateway uses PostgreSQL for persistent storage. Configure the connection string:

DATABASE_URL=postgres://user:password@host:port/database

For production, use connection pooling and SSL:

DATABASE_URL=postgres://user:password@host:port/database?sslmode=require&max_connections=20

Redis

Redis is used for response caching and session storage:

REDIS_URL=redis://host:port
# Or with authentication:
REDIS_URL=redis://user:password@host:port

Enable Redis persistence (AOF or RDB) in production to avoid losing cache data on restarts.

Provider API keys

Add API keys for the LLM providers you want to use:

# OpenAI
OPENAI_API_KEY=sk-...

# Anthropic
ANTHROPIC_API_KEY=sk-ant-...

# Google Vertex AI
GOOGLE_SERVICE_ACCOUNT_JSON={"type": "service_account", ...}

# AWS Bedrock
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_REGION=us-east-1

See environment variables reference for all supported providers.

Authentication

Configure Better Auth for user authentication:

BETTER_AUTH_SECRET=$(openssl rand -hex 32)
BETTER_AUTH_URL=https://your-domain.com

# Optional: Enable GitHub OAuth
GITHUB_CLIENT_ID=...
GITHUB_CLIENT_SECRET=...

# Optional: Enable Google OAuth
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...

Health checks

Monitor the health of your deployment using these endpoints:

# Gateway health (includes database and Redis checks)
curl http://localhost:4001/

# API health
curl http://localhost:4002/api/health

Response format:

{
  "message": "LLMGateway API is running",
  "version": "1.0.0",
  "health": {
    "status": "healthy",
    "redis": {
      "connected": true
    },
    "database": {
      "connected": true
    }
  }
}

If either Redis or the database is unavailable, the health check returns 503 Service Unavailable.

Monitoring

Prometheus metrics

LLM Gateway exposes Prometheus metrics at /metrics:

curl http://localhost:4001/metrics

Metrics include:

Request counts and durations
Token usage by model and provider
Cache hit/miss rates
Error rates
Database connection pool stats

OpenTelemetry

Enable distributed tracing with OpenTelemetry:

OTEL_ENABLED=true
OTEL_SERVICE_NAME=llmgateway
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318

Backup and restore

Database backup

Back up the PostgreSQL database regularly:

# Create backup
docker compose exec postgres pg_dump -U postgres llmgateway > backup.sql

# Restore backup
docker compose exec -T postgres psql -U postgres llmgateway < backup.sql

Redis backup

Redis persistence is optional but recommended for production. Enable AOF or RDB snapshots in your Redis configuration.

Troubleshooting

Database connection errors

If you see database connection errors:

Verify PostgreSQL is running: docker compose ps postgres
Check the connection string in .env
Ensure the database exists: docker compose exec postgres psql -U postgres -l
Check database logs: docker compose logs postgres

Redis connection errors

If Redis is unavailable:

Verify Redis is running: docker compose ps redis
Check the connection string in .env
Test connectivity: docker compose exec redis redis-cli ping

Provider API errors

If LLM requests fail:

Verify provider API keys are set correctly in .env
Check provider key status in the UI dashboard
Review request logs: docker compose logs gateway
Test directly with the provider’s API to rule out key issues

Build errors

If build fails:

Ensure Node.js version is 20 or later: node --version
Clear build cache: pnpm clean
Reinstall dependencies: rm -rf node_modules && pnpm install
Check for TypeScript errors: pnpm build:core

Next steps

Environment variables

Complete reference for all configuration options

Docker deployment

Production-ready Docker configuration

Kubernetes deployment

Deploy to Kubernetes with Helm charts

Security best practices

Secure your deployment for production use

Get Started

Core Features

Guides

Integrations

Prerequisites

Quick start with Docker Compose

Docker Compose configuration

Building from source

Production deployment

Docker deployment

Kubernetes deployment

Configuration

Database

Redis

Provider API keys

Authentication

Health checks

Monitoring

Prometheus metrics

OpenTelemetry

Backup and restore

Database backup

Redis backup

Troubleshooting

Database connection errors

Redis connection errors

Provider API errors

Build errors

Next steps

Environment variables

Docker deployment

Kubernetes deployment

Security best practices

Build docs developers (and LLMs) love

Get Started

Core Features

Guides

Integrations

​Prerequisites

​Quick start with Docker Compose

​Docker Compose configuration

​Building from source

​Production deployment

Docker deployment

Kubernetes deployment

​Configuration

​Database

​Redis

​Provider API keys

​Authentication

​Health checks

​Monitoring

​Prometheus metrics

​OpenTelemetry

​Backup and restore

​Database backup

​Redis backup

​Troubleshooting

​Database connection errors

​Redis connection errors

​Provider API errors

​Build errors

​Next steps

Environment variables

Docker deployment

Kubernetes deployment

Security best practices

Build docs developers (and LLMs) love

Prerequisites

Quick start with Docker Compose

Docker Compose configuration

Building from source

Production deployment

Configuration

Database

Redis

Provider API keys

Authentication

Health checks

Monitoring

Prometheus metrics

OpenTelemetry

Backup and restore

Database backup

Redis backup

Troubleshooting

Database connection errors

Redis connection errors

Provider API errors

Build errors

Next steps