AI Gateway

AI Gateway provides a unified interface for AI providers with caching, rate limiting, analytics, and cost controls. Use the API to create gateways and manage AI traffic.

Overview

Access the AI Gateway API:

import Cloudflare from 'cloudflare';

const client = new Cloudflare({
  apiToken: process.env.CLOUDFLARE_API_TOKEN,
});

// Access AI Gateway resources
const aiGateway = client.aiGateway;

Gateways

Manage AI Gateway instances.

Create a gateway

Create a new AI Gateway.

const gateway = await client.aiGateway.create({
  account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
  id: 'my-gateway',
  cache_invalidate_on_update: true,
  cache_ttl: 3600,
  collect_logs: true,
  rate_limiting_interval: 60,
  rate_limiting_limit: 100,
  rate_limiting_technique: 'sliding',
});

account_id

string

required

Your Cloudflare account ID

string

required

Gateway identifier (used in API calls)

cache_ttl

number

Cache time-to-live in seconds (null to disable caching)

cache_invalidate_on_update

boolean

required

Invalidate cache when gateway is updated

collect_logs

boolean

required

Enable request/response logging

rate_limiting_interval

number

Rate limit window in seconds (null to disable)

rate_limiting_limit

number

Maximum requests per interval (null to disable)

rate_limiting_technique

string

required

Rate limiting algorithm: ‘fixed’ or ‘sliding’

string

The gateway ID

account_id

string

Your account ID

created_at

string

ISO 8601 timestamp when the gateway was created

modified_at

string

ISO 8601 timestamp when the gateway was last modified

List gateways

Retrieve all AI Gateways in your account.

for await (const gateway of client.aiGateway.list({
  account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
})) {
  console.log(gateway);
}

account_id

string

required

Your Cloudflare account ID

string

Filter gateways by ID

page

number

Page number for pagination

per_page

number

Number of gateways per page

Get a gateway

Retrieve details about a specific gateway.

const gateway = await client.aiGateway.get(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

gateway_id

string

required

The gateway ID

account_id

string

required

Your Cloudflare account ID

Update a gateway

Update gateway configuration.

const gateway = await client.aiGateway.update(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    cache_ttl: 7200,
    rate_limiting_limit: 200,
    rate_limiting_interval: 60,
    rate_limiting_technique: 'sliding',
    cache_invalidate_on_update: true,
    collect_logs: true,
  }
);

gateway_id

string

required

The gateway ID to update

account_id

string

required

Your Cloudflare account ID

Delete a gateway

Delete an AI Gateway.

await client.aiGateway.delete(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

gateway_id

string

required

The gateway ID to delete

account_id

string

required

Your Cloudflare account ID

Logs

Manage and query AI Gateway request logs.

List logs

Retrieve logs for a gateway.

for await (const log of client.aiGateway.logs.list(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    start_date: '2024-01-01',
    end_date: '2024-01-31',
  }
)) {
  console.log(log);
}

gateway_id

string

required

The gateway ID

account_id

string

required

Your Cloudflare account ID

start_date

string

Start date for logs (ISO 8601)

end_date

string

End date for logs (ISO 8601)

page

number

Page number for pagination

per_page

number

Number of logs per page

Get a log entry

Retrieve a specific log entry.

const log = await client.aiGateway.logs.get(
  'my-gateway',
  logId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

string

The log entry ID

timestamp

string

ISO 8601 timestamp of the request

provider

string

AI provider (e.g., ‘openai’, ‘anthropic’)

model

string

Model used for the request

status_code

number

HTTP status code

cached

boolean

Whether the response was served from cache

duration_ms

number

Request duration in milliseconds

Get log request

Retrieve the request payload for a log entry.

const request = await client.aiGateway.logs.request(
  'my-gateway',
  logId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Get log response

Retrieve the response payload for a log entry.

const response = await client.aiGateway.logs.response(
  'my-gateway',
  logId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Delete logs

Delete logs for a gateway.

await client.aiGateway.logs.delete(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    before: '2024-01-01',
  }
);

before

string

Delete logs before this date (ISO 8601)

Datasets

Manage datasets for AI model evaluation.

Create a dataset

const dataset = await client.aiGateway.datasets.create(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    name: 'test-dataset',
    data: [
      { input: 'Hello', expected_output: 'Hi there!' },
      { input: 'Goodbye', expected_output: 'See you later!' },
    ],
  }
);

gateway_id

string

required

The gateway ID

account_id

string

required

Your Cloudflare account ID

name

string

required

Dataset name

data

array

required

Array of test cases with input and expected output

List datasets

for await (const dataset of client.aiGateway.datasets.list(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
)) {
  console.log(dataset);
}

Get a dataset

const dataset = await client.aiGateway.datasets.get(
  'my-gateway',
  datasetId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Update a dataset

const dataset = await client.aiGateway.datasets.update(
  'my-gateway',
  datasetId,
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    name: 'updated-dataset',
  }
);

Delete a dataset

await client.aiGateway.datasets.delete(
  'my-gateway',
  datasetId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

Evaluations

Run model evaluations using datasets.

Create an evaluation

const evaluation = await client.aiGateway.evaluations.create(
  'my-gateway',
  {
    account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0',
    dataset_id: datasetId,
    evaluation_type: 'exact_match',
  }
);

gateway_id

string

required

The gateway ID

account_id

string

required

Your Cloudflare account ID

dataset_id

string

required

Dataset to evaluate against

evaluation_type

string

required

Evaluation metric type

List evaluations

for await (const evaluation of client.aiGateway.evaluations.list(
  'my-gateway',
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
)) {
  console.log(evaluation);
}

Get an evaluation

const evaluation = await client.aiGateway.evaluations.get(
  'my-gateway',
  evaluationId,
  { account_id: '3ebbcb006d4d46d7bb6a8c7f14676cb0' }
);

string

Evaluation ID

status

string

Evaluation status: ‘pending’, ‘running’, or ‘complete’

score

number

Evaluation score (0-1)

results

array

Detailed results per test case

Using AI Gateway

With OpenAI SDK

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: 'https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai',
});

const completion = await openai.chat.completions.create({
  model: 'gpt-3.5-turbo',
  messages: [{ role: 'user', content: 'Hello!' }],
});

With Workers AI

const response = await fetch(
  'https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/workers-ai/@cf/meta/llama-2-7b-chat-int8',
  {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${CLOUDFLARE_API_TOKEN}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      prompt: 'Tell me a joke',
    }),
  }
);

Best practices

Caching: Enable caching for repeated queries to reduce costs and latency
Rate limiting: Set appropriate rate limits to control costs
Logging: Enable logging for debugging and analytics
Monitoring: Regularly review logs and analytics for usage patterns
Security: Use authentication to protect your gateway from unauthorized access
Cost control: Set up alerts for spending thresholds

Core Resources

Workers & Serverless

Application Services

Security & Access

Network Services

Developer Platform

Management & Monitoring

Overview

Gateways

Create a gateway

List gateways

Get a gateway

Update a gateway

Delete a gateway

Logs

List logs

Get a log entry

Get log request

Get log response

Delete logs

Datasets

Create a dataset

List datasets

Get a dataset

Update a dataset

Delete a dataset

Evaluations

Create an evaluation

List evaluations

Get an evaluation

Using AI Gateway

With OpenAI SDK

With Workers AI

Best practices

Build docs developers (and LLMs) love

Core Resources

Workers & Serverless

Application Services

Security & Access

Network Services

Developer Platform

Management & Monitoring

​Overview

​Gateways

​Create a gateway

​List gateways

​Get a gateway

​Update a gateway

​Delete a gateway

​Logs

​List logs

​Get a log entry

​Get log request

​Get log response

​Delete logs

​Datasets

​Create a dataset

​List datasets

​Get a dataset

​Update a dataset

​Delete a dataset

​Evaluations

​Create an evaluation

​List evaluations

​Get an evaluation

​Using AI Gateway

​With OpenAI SDK

​With Workers AI

​Best practices

Build docs developers (and LLMs) love

Overview

Gateways

Create a gateway

List gateways

Get a gateway

Update a gateway

Delete a gateway

Logs

List logs

Get a log entry

Get log request

Get log response

Delete logs

Datasets

Create a dataset

List datasets

Get a dataset

Update a dataset

Delete a dataset

Evaluations

Create an evaluation

List evaluations

Get an evaluation

Using AI Gateway

With OpenAI SDK

With Workers AI

Best practices