Skip to main content

​
Quickstart

Get up and running with LLM Gateway in just a few minutes. This guide will take you from installation to making your first LLM call with tools.

​
What You’ll Build

By the end of this quickstart, you’ll have:
  • A working LLM Gateway setup
  • Your first streaming LLM call
  • An agent that can execute shell commands
  • Understanding of how to add custom tools

​
Prerequisites

1
Install LLM Gateway
2
Clone the repository and install dependencies:
3
git clone https://github.com/yourusername/llm-gateway.git
cd llm-gateway
bun install
4
Configure Your Environment
5
Copy the example environment file and add your API key:
6
cp .env.example .env
7
Edit .env and add your API key:
8
ZEN_API_KEY=your_api_key_here
DEFAULT_MODEL=glm-4.7
LOG_LEVEL=I
9
You can use OPENROUTER_API_KEY or ANTHROPIC_API_KEY instead, depending on your provider.
10
Make Your First LLM Call
11
Create a new file called example.ts in the root directory:
12
import { createGeneratorHarness } from "./packages/ai/harness/providers/zen";

const harness = createGeneratorHarness();

for await (const event of harness.invoke({
  model: "glm-4.7",
  messages: [{ role: "user", content: "What is the sum of the first 10 primes?" }],
})) {
  if (event.type === "reasoning") process.stderr.write(event.content); // thinking
  if (event.type === "text") process.stdout.write(event.content);
}
13
Run it:
14
bun run example.ts
15
You should see the model’s response streaming to your terminal!
16
Add Tool Calling
17
Now let’s create an agent that can execute shell commands. Update your example.ts:
18
import { createAgentHarness } from "./packages/ai/harness/agent";
import { createGeneratorHarness } from "./packages/ai/harness/providers/zen";
import { bashTool } from "./packages/ai/tools";

const agent = createAgentHarness({ harness: createGeneratorHarness() });

for await (const event of agent.invoke({
  model: "glm-4.7",
  messages: [{ role: "user", content: "List the files in this directory" }],
  tools: [bashTool],
  permissions: { allowlist: [{ tool: "bash" }] },
})) {
  if (event.type === "reasoning") process.stderr.write(event.content);
  if (event.type === "text") process.stdout.write(event.content);
  if (event.type === "tool_call") console.log(`\n[calling ${event.name}]`);
  if (event.type === "tool_result") console.log(`[result]`, event.output);
}
19
Run it again:
20
bun run example.ts
21
Now the agent will call the bash tool to execute ls and use the result to answer your question!
22
The permissions parameter controls which tools the agent can use. Use allowlist for auto-approval, or omit it for human-in-the-loop approval.
23
Start the Full Server
24
For a production setup with HTTP/SSE endpoints:
25
bun run dev:server
26
The server will start on http://localhost:4000 with these endpoints:
27
  • GET /models - List available models
  • POST /chat - SSE streaming chat endpoint
  • POST /chat/relay/:relayId - Permission resolution endpoint

    • ​
    Understanding Events

    LLM Gateway is built around a simple event-driven architecture. Every harness yields events as an async generator: 
    // Core streaming events
{ type: "text", content: "Hello" }           // Streamed text from the model
{ type: "reasoning", content: "thinking..." } // Model's reasoning process

// Tool execution events
{ type: "tool_call", name: "bash", input: { command: "ls" } }
{ type: "tool_result", output: { exitCode: 0, stdout: "..." } }

// Lifecycle events
{ type: "harness_start" }  // Agent loop begins
{ type: "harness_end" }    // Agent loop completes
{ type: "usage" }          // Token usage information

// Permission events
{ type: "relay", kind: "permission", tool: "bash", ... }
    Every event includes:
    • runId - Which LLM call produced this event
    • parentId - Which run spawned this one (for subagents)

    ​
    Next Steps

    Add Custom Tools

    Learn how to create your own tools for agents to use

    Multi-Agent Systems

    Build systems with multiple concurrent agents

    Client Integration

    Build UIs that consume agent events

    API Reference

    Explore the full HTTP API

    ​
    Common Patterns

    ​
    Choosing a Provider

    LLM Gateway supports multiple LLM providers: 
    // Zen (default, fastest, supports reasoning)
import { createGeneratorHarness } from "./packages/ai/harness/providers/zen";

// Anthropic (Claude models)
import { createGeneratorHarness } from "./packages/ai/harness/providers/anthropic";

// OpenAI
import { createGeneratorHarness } from "./packages/ai/harness/providers/openai";

// OpenRouter (100+ models)
import { createGeneratorHarness } from "./packages/ai/harness/providers/openrouter";

    ​
    Tool Permissions

    Control tool execution with fine-grained permissions: 
    // Auto-approve all bash commands
permissions: {
  allowlist: [{ tool: "bash" }]
}

// Require approval for each tool call
permissions: {} // No allowlist

// Auto-approve specific commands only
permissions: {
  allowlist: [
    { tool: "bash", params: { command: "ls **" } },
    { tool: "bash", params: { command: "git **" } }
  ]
}

// Explicitly deny a tool call
permissions: {
  deny: [{ toolCallId: "call_123", reason: "Not allowed" }]
}

    ​
    Human-in-the-Loop

    Pause execution for human approval: 
    import { AgentOrchestrator } from "./packages/ai/orchestrator";
import { createAgentHarness } from "./packages/ai/harness/agent";
import { createGeneratorHarness } from "./packages/ai/harness/providers/zen";
import { bashTool } from "./packages/ai/tools";

const orchestrator = new AgentOrchestrator(
  createAgentHarness({ harness: createGeneratorHarness() })
);

orchestrator.spawn({
  model: "glm-4.7",
  messages: [{ role: "user", content: "Delete all logs" }],
  tools: [bashTool],
  // No allowlist - will require approval
});

for await (const { agentId, event } of orchestrator.events()) {
  if (event.type === "relay") {
    // Ask the user
    const approved = await askUser(`Allow ${event.tool}?`);
    orchestrator.resolveRelay(event.id, { approved, always: false });
  }
  if (event.type === "text") process.stdout.write(event.content);
}

    ​
    Troubleshooting

    Make sure you set DEFAULT_MODEL in your .env file or pass the model parameter to invoke().
    Check that your .env file has the correct API key variable:
    • ZEN_API_KEY for Zen
    • OPENROUTER_API_KEY for OpenRouter
    • ANTHROPIC_API_KEY for Anthropic
    Make sure you:
    1. Passed tools: [...] to invoke()
    2. Used createAgentHarness() not just the provider harness
    3. Set appropriate permissions
    Ensure you’re using for await to consume the async generator:
    for await (const event of harness.invoke(...)) {
  // Handle events
}

    Build docs developers (and LLMs) love

    Get started for freeTalk to us