Installation

This guide covers installing LLM Gateway, configuring your environment, and understanding the project structure.

System Requirements

Runtime

Bun 1.0+ required
Install Bun →

Operating System

macOS, Linux, or WSL2
Windows native not tested

Node Version

Not required
Bun replaces Node.js

Memory

2GB+ RAM recommended
For running local models

Installation Steps

Clone the Repository

git clone https://github.com/yourusername/llm-gateway.git
cd llm-gateway

Install Dependencies

LLM Gateway uses Bun as both runtime and package manager:

bun install

This installs all dependencies from package.json:

View Core Dependencies

Runtime & Server:

hono - Web framework for HTTP/SSE endpoints
effect - Error handling and retries
uuid - Generate unique IDs for events

LLM Providers:

@anthropic-ai/sdk - Anthropic Claude integration
openai - OpenAI and compatible APIs
@openrouter/sdk - OpenRouter integration

Tools & Utilities:

picomatch - Glob pattern matching for permissions
zod - Schema validation and JSON Schema generation
oxfmt - Code formatter

Client Libraries:

react / react-dom - Web client UI
@vitejs/plugin-react - Build tooling

Configure Environment

Copy the example environment file:

cp .env.example .env

Edit .env with your configuration:

# API Keys - Add at least one
ZEN_API_KEY=your_zen_api_key_here
OPENROUTER_API_KEY=
ANTHROPIC_API_KEY=

# Model Configuration
DEFAULT_MODEL=glm-4.7

# Logging
LOG_LEVEL=I  # I=Info, W=Warn, E=Error

# Server Configuration
PORT=4000
VITE_PORT=4001
VITE_BACKEND_URL=http://localhost:4000

You only need one API key to get started. Choose based on your preferred provider:

Zen: Fastest, supports reasoning, free tier available
OpenRouter: 100+ models, pay-per-use
Anthropic: Direct access to Claude models

Verify Installation

Test your setup by running the development server:

bun run dev:server

You should see:

LLM Gateway server running on http://localhost:4000

Test the API:

curl http://localhost:4000/models

Expected response:

{
  "models": [
    "glm-4.7",
    "nvidia/nemotron-nano-9b-v2:free",
    ...
  ],
  "defaultModel": "glm-4.7"
}

Project Structure

Understanding the codebase structure will help you navigate and extend LLM Gateway:

llm-gateway/
├── packages/ai/           # Core AI orchestration engine
│   ├── harness/          # Provider and agent harnesses
│   │   ├── providers/   # LLM provider adapters
│   │   │   ├── zen.ts      # Zen (default, fastest)
│   │   │   ├── anthropic.ts
│   │   │   ├── openai.ts
│   │   │   └── openrouter.ts
│   │   └── agent.ts     # Agentic wrapper with tools
│   ├── tools/           # Built-in tools
│   │   ├── bash.ts         # Shell command execution
│   │   ├── read.ts         # File reading
│   │   ├── patch.ts        # File patching
│   │   └── agent.ts        # Subagent spawning
│   ├── client/          # Client-side state management
│   │   ├── hypergraph/     # Event graph reducer
│   │   ├── transports/     # SSE and HTTP transports
│   │   └── projections/    # Graph-to-UI projections
│   ├── rlm/             # Recursive Language Model
│   ├── primitives/      # Async coordination utilities
│   ├── orchestrator.ts  # Multi-agent orchestration
│   ├── permissions.ts   # Tool permission system
│   └── types.ts         # Core type definitions
├── server/              # HTTP/SSE server
│   └── index.ts           # Hono app with endpoints
├── clients/             # Client applications
│   ├── cli/              # Terminal UI client
│   └── web/              # React web client
├── .env.example         # Environment template
└── package.json         # Dependencies and scripts

Key Modules Explained

Harnesses (packages/ai/harness/)The fundamental primitive - async generators that yield events:

Provider harnesses make single LLM API calls
Agent harness wraps providers to add tool execution loop
Harnesses compose: createAgentHarness({ harness: createGeneratorHarness() })

Tools (packages/ai/tools/)Executable capabilities for agents:

Each tool has a Zod schema, description, and execute function
Tools return { context, result } - context goes to the agent, result to your code
Permissions derived from tool parameters using glob patterns

Client (packages/ai/client/)Client-side state management:

Events → Hypergraph → Projections → UI
projectThread() for chat UI, projectMessages() for API format
SSE transport for streaming, HTTP transport for relay resolution

Orchestrator (orchestrator.ts)Manages multiple concurrent agents:

Multiplexes agent streams via AgentMultiplexer
Handles permission relays between agents and humans
Automatic cleanup when agents complete

Development Commands

LLM Gateway provides several npm scripts for development:

Server Development

# Start server with hot reload
bun run dev:server

# Server runs on http://localhost:4000
# Changes auto-reload with Bun's --hot flag

The server provides:

GET /models - List available models
POST /chat - SSE streaming chat endpoint
POST /chat/relay/:relayId - Permission resolution
POST /summarize - Conversation summarization

Web Client Development

# Start web client with Vite
bun run dev:web

# Opens on http://localhost:4001
# Hot module replacement enabled

CLI Client

# Run terminal UI client
bun run dev:cli

# Interactive TUI with streaming responses
# Uses SolidJS for fine-grained reactivity

Testing

# Run all tests
bun test

# Run specific test file
bun test packages/ai/harness/__tests__/agent.test.ts

# Tests use Bun's built-in test runner
# No mocks - real integrations only

Tests only output on failure (quiet success, loud failure). This is intentional - see Development Principles in CLAUDE.md.

Code Formatting

# Format all code with oxfmt
bun run format

# Formats src/, packages/, clients/, server/

Build for Production

# Build web client
bun run build:web

# Output in clients/web/dist/
# Ready for static hosting

Environment Variables Reference

Complete list of environment variables:

ZEN_API_KEY

string

API key for Zen provider (OpenAI-compatible with reasoning support)

OPENROUTER_API_KEY

string

API key for OpenRouter (access to 100+ models)

ANTHROPIC_API_KEY

string

API key for Anthropic Claude models

DEFAULT_MODEL

string

default:"glm-4.7"

Default model to use when none specified in requests

LOG_LEVEL

string

default:"I"

Logging verbosity: I (Info), W (Warn), E (Error)

PORT

number

default:"4000"

Server port for HTTP/SSE endpoints

VITE_PORT

number

default:"4001"

Vite dev server port for web client

VITE_BACKEND_URL

string

default:"http://localhost:4000"

Backend URL for web client to connect to

Provider Setup

Detailed setup for each LLM provider:

Zen
OpenRouter
Anthropic
OpenAI

Zen is the default provider, optimized for speed and supporting reasoning tokens.Get an API Key:

Visit opencode.ai/zen
Sign up for an account
Generate an API key

Configure:

.env

ZEN_API_KEY=your_key_here
DEFAULT_MODEL=glm-4.7

Supported Models:

glm-4.7 - Fast, supports reasoning
nvidia/nemotron-nano-9b-v2:free - Free tier
kimi-k2.5 - Large context window

Usage:

import { createGeneratorHarness } from "./packages/ai/harness/providers/zen";

const harness = createGeneratorHarness({
  apiKey: process.env.ZEN_API_KEY,
  model: "glm-4.7"
});

OpenRouter provides access to 100+ models from multiple providers.Get an API Key:

Visit openrouter.ai
Sign up and add credits
Generate an API key

Configure:

.env

OPENROUTER_API_KEY=your_key_here
DEFAULT_MODEL=anthropic/claude-3.5-sonnet

Popular Models:

anthropic/claude-3.5-sonnet - Best reasoning
google/gemini-2.0-flash-exp:free - Fast, free
meta-llama/llama-3.3-70b-instruct - Open source

Usage:

import { createGeneratorHarness } from "./packages/ai/harness/providers/openrouter";

const harness = createGeneratorHarness({
  apiKey: process.env.OPENROUTER_API_KEY
});

Direct access to Claude models via Anthropic’s API.Get an API Key:

Visit console.anthropic.com
Create an account and add credits
Generate an API key

Configure:

.env

ANTHROPIC_API_KEY=your_key_here
DEFAULT_MODEL=claude-3-5-sonnet-20241022

Available Models:

claude-3-5-sonnet-20241022 - Latest Sonnet
claude-3-5-haiku-20241022 - Fastest Claude
claude-3-opus-20240229 - Most capable

Usage:

import { createGeneratorHarness } from "./packages/ai/harness/providers/anthropic";

const harness = createGeneratorHarness({
  apiKey: process.env.ANTHROPIC_API_KEY
});

Use OpenAI models or any OpenAI-compatible API.Get an API Key:

Visit platform.openai.com
Create an account and add credits
Generate an API key

Configure:

.env

OPENAI_API_KEY=your_key_here
DEFAULT_MODEL=gpt-4o

Available Models:

gpt-4o - Latest multimodal
gpt-4o-mini - Fast and cheap
o1 - Advanced reasoning

Usage:

import { createGeneratorHarness } from "./packages/ai/harness/providers/openai";

const harness = createGeneratorHarness({
  apiKey: process.env.OPENAI_API_KEY
});

Docker Setup (Optional)

Run LLM Gateway in a container:

Dockerfile

FROM oven/bun:1

WORKDIR /app

COPY package.json bun.lockb ./
RUN bun install --frozen-lockfile

COPY . .

EXPOSE 4000

CMD ["bun", "run", "server/index.ts"]

Build and run:

# Build image
docker build -t llm-gateway .

# Run container
docker run -p 4000:4000 --env-file .env llm-gateway

Or use Docker Compose:

docker-compose.yml

version: '3.8'

services:
  server:
    build: .
    ports:
      - "4000:4000"
    env_file:
      - .env
    volumes:
      - ./data:/app/data

Run with:

docker-compose up

Troubleshooting

Bun installation fails

Issue: curl -fsSL https://bun.sh/install | bash failsSolutions:

Use the manual installation: Download from bun.sh/install
On macOS: brew install bun
Check system requirements: macOS 11+, Linux kernel 5.1+
For WSL2: Make sure you’re using Ubuntu 20.04+

Dependencies fail to install

Issue: bun install errors or hangsSolutions:

Clear Bun cache: rm -rf ~/.bun/install/cache
Update Bun: bun upgrade
Check internet connection and firewall
Try with --verbose flag: bun install --verbose

Server won't start

Issue: bun run dev:server failsSolutions:

Check if port 4000 is in use: lsof -i :4000
Verify .env file exists and has valid API key
Check logs for specific error messages
Try different port: PORT=3000 bun run dev:server

API calls return 401 Unauthorized

Issue: LLM provider returns authentication errorSolutions:

Verify API key is correct in .env
Check key hasn’t expired or been revoked
Ensure no extra spaces or quotes in .env file
Test key with provider’s curl example

Tests fail

Issue: bun test shows failuresSolutions:

Ensure API keys are set (tests use real integrations)
Check if provider APIs are accessible
Verify no rate limiting is occurring
Run individual test files to isolate issue

Next Steps

Quickstart Guide

Build your first agent in 5 minutes

Core Concepts

Understand harnesses, events, and composition

API Reference

Explore HTTP endpoints and types

Tool Development

Create custom tools for your agents

Get Started

Core Concepts

Guides

Building Extensions

Deployment

Installation

Installation

System Requirements

Runtime

Operating System

Node Version

Memory

Installation Steps

Project Structure

Development Commands

Server Development

Web Client Development

CLI Client

Testing

Code Formatting

Build for Production

Environment Variables Reference

Provider Setup

Docker Setup (Optional)

Troubleshooting

Next Steps

Quickstart Guide

Core Concepts

API Reference

Tool Development

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Building Extensions

Deployment

​Installation

​System Requirements

Runtime

Operating System

Node Version

Memory

​Installation Steps

​Project Structure

​Development Commands

​Server Development

​Web Client Development

​CLI Client

​Testing

​Code Formatting

​Build for Production

​Environment Variables Reference

​Provider Setup

​Docker Setup (Optional)

​Troubleshooting

​Next Steps

Quickstart Guide

Core Concepts

API Reference

Tool Development

Build docs developers (and LLMs) love

Installation

System Requirements

Installation Steps

Project Structure

Development Commands

Server Development

Web Client Development

CLI Client

Testing

Code Formatting

Build for Production

Environment Variables Reference

Provider Setup

Docker Setup (Optional)

Troubleshooting

Next Steps