Configs - Portkey AI Gateway

Configs are the core mechanism for controlling how the Portkey AI Gateway processes requests. They define routing rules, retry policies, caching behavior, guardrails, and more.

Config Structure

A config is a JSON object that can be passed via headers or in the request body. Configs support both snake_case and camelCase keys.

Basic Config

{
  "provider": "openai",
  "api_key": "sk-..."
}

Complete Config Example

{
  "strategy": {
    "mode": "fallback",
    "on_status_codes": [429, 500, 502, 503]
  },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-...",
      "retry": {
        "attempts": 3,
        "on_status_codes": [429, 500]
      },
      "request_timeout": 30000,
      "cache": {
        "mode": "simple",
        "max_age": 3600000
      },
      "input_guardrails": [
        {
          "default.contains": {
            "operator": "none",
            "words": ["sensitive-info"]
          },
          "deny": true
        }
      ],
      "output_guardrails": [
        {
          "default.contains": {
            "operator": "none",
            "words": ["restricted"]
          },
          "deny": true
        }
      ]
    },
    {
      "provider": "anthropic",
      "api_key": "sk-ant-..."
    }
  ]
}

Config Fields

Provider Configuration

Field	Type	Description
`provider`	string	Provider name (e.g., “openai”, “anthropic”, “azure-openai”)
`api_key`	string	Provider API key
`virtual_key`	string	Portkey virtual key for secure key management
`custom_host`	string	Custom endpoint URL for provider

Strategy Configuration

Field	Type	Description
`strategy.mode`	string	Routing mode: “single”, “fallback”, “loadbalance”, “conditional”
`strategy.on_status_codes`	number[]	Status codes that trigger fallback (fallback mode only)
`strategy.conditions`	array	Conditional routing rules (conditional mode only)
`strategy.default`	string	Default target for conditional routing

Retry Configuration

Field	Type	Description
`retry.attempts`	number	Maximum retry attempts (0-5)
`retry.on_status_codes`	number[]	Status codes that trigger retry
`retry.use_retry_after_header`	boolean	Respect provider’s Retry-After header (default: false)

Retry attempts are capped at 5 to prevent excessive retry loops. The gateway uses exponential backoff between retries.

See Retries for detailed retry configuration.

Cache Configuration

Field	Type	Description
`cache.mode`	string	”simple” or “semantic”
`cache.max_age`	number	Cache TTL in milliseconds

Streaming requests are not cached. The cache middleware automatically skips streaming requests.

See Caching for detailed cache configuration.

Timeout Configuration

Field	Type	Description
`request_timeout`	number	Request timeout in milliseconds

When a request exceeds the timeout, the gateway returns a 408 status with a timeout error. Source: src/handlers/retryHandler.ts:4-50

Guardrails Configuration

Field	Type	Description
`input_guardrails`	array	Input validation rules executed before provider request
`output_guardrails`	array	Output validation rules executed after provider response
`before_request_hooks`	array	Custom hooks executed before request
`after_request_hooks`	array	Custom hooks executed after response

See Guardrails for detailed guardrail configuration.

Provider-Specific Fields

Azure OpenAI

{
  "provider": "azure-openai",
  "api_key": "...",
  "azure_model_name": "gpt-4",
  "azure_auth_mode": "api_key",
  "resource_name": "your-resource",
  "deployment_id": "your-deployment",
  "api_version": "2024-02-15-preview"
}

Google Vertex AI

{
  "provider": "vertex-ai",
  "vertex_project_id": "your-project-id",
  "vertex_region": "us-central1",
  "api_key": "ya29...",
  "vertex_service_account_json": {
    "type": "service_account",
    "project_id": "..."
  }
}

Vertex AI requires either vertex_project_id or vertex_service_account_json along with vertex_region.

AWS Bedrock

{
  "provider": "bedrock",
  "aws_access_key_id": "AKIA...",
  "aws_secret_access_key": "...",
  "aws_region": "us-east-1",
  "aws_session_token": "..." // Optional
}

OpenAI

{
  "provider": "openai",
  "api_key": "sk-...",
  "openai_organization": "org-...",
  "openai_project": "proj_..."
}

Config Inheritance

Configs support inheritance where child targets inherit properties from parent configs:

{
  "retry": { "attempts": 3 },
  "request_timeout": 30000,
  "strategy": { "mode": "loadbalance" },
  "targets": [
    {
      "provider": "openai",
      "api_key": "sk-1"
      // Inherits retry and request_timeout from parent
    },
    {
      "provider": "openai",
      "api_key": "sk-2",
      "retry": { "attempts": 5 }
      // Overrides parent retry attempts
    }
  ]
}

Inheritance Rules:

Child configs override parent configs for the same field
Arrays (like hooks and guardrails) are merged, not replaced
Nested targets can define their own strategies

Source: src/handlers/handlerUtils.ts:470-644

Config Validation

The gateway validates all configs using Zod schemas. Invalid configs are rejected with a 400 status code.

Validation Rules

Provider + API Key OR Strategy + Targets - Configs must have either:
- A provider and api_key, OR
- A strategy with targets, OR
- Cache/retry/timeout settings
Valid Provider - Provider must be in the list of supported providers
Valid Strategy Mode - Must be one of: single, loadbalance, fallback, conditional
Valid Cache Mode - Must be “simple” or “semantic”
Custom Host Format - Must be a valid URL if provided

Source: src/middlewares/requestValidator/schema/config.ts:11-179

Passing Configs

Via Header

curl https://gateway.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-portkey-config: $(cat config.json | jq -c)" \
  -d '{...}'

Via SDK

from portkey_ai import Portkey

config = {
    "strategy": {"mode": "fallback"},
    "targets": [
        {"provider": "openai", "api_key": "sk-..."},
        {"provider": "anthropic", "api_key": "sk-ant-..."}
    ]
}

client = Portkey(config=config)

Via Environment

Configs can also be stored in conf.json at the gateway root for system-wide defaults.

{
  "cache": false,
  "plugins_enabled": ["default", "portkey"],
  "integrations": [
    {
      "provider": "anthropic",
      "slug": "dev_team_anthropic",
      "credentials": {
        "apiKey": "sk-ant-"
      }
    }
  ]
}

Config Conversion

The gateway automatically converts between snake_case and camelCase:

// Both formats work identically
const config1 = { "api_key": "sk-...", "request_timeout": 30000 }
const config2 = { "apiKey": "sk-...", "requestTimeout": 30000 }

Source: src/handlers/handlerUtils.ts:1007-1165

Override Parameters

You can override request parameters using override_params:

{
  "provider": "openai",
  "api_key": "sk-...",
  "override_params": {
    "model": "gpt-4-turbo",
    "temperature": 0.7,
    "max_tokens": 1000
  }
}

Override params are merged with the request body, with override params taking precedence.

Forward Headers

Specify which request headers should be forwarded to the provider:

{
  "provider": "openai",
  "api_key": "sk-...",
  "forward_headers": [
    "user-agent",
    "x-custom-header"
  ]
}

The gateway automatically filters out Portkey-specific headers (prefixed with x-portkey-) from forwarded headers.

OpenAI Compliance

For strict OpenAI API compliance, set strict_open_ai_compliance:

{
  "provider": "anthropic",
  "api_key": "sk-ant-...",
  "strict_open_ai_compliance": true
}

This removes provider-specific fields from responses that aren’t part of the OpenAI spec.

Best Practices

Version Your Configs

Store configs in version control to track changes and enable rollbacks.

Use Virtual Keys

Use virtual keys instead of hardcoding API keys for better security.

Test Configs Locally

Test config changes locally before deploying to production.

Monitor Config Performance

Track metrics for different configs to optimize routing and retries.

Next Steps

Routing

Learn about routing strategies and modes.

Guardrails

Configure input and output validation.

Caching

Optimize performance with response caching.

Load Balancing

Distribute load across providers and keys.

Getting Started

Core Concepts

Features

MCP Gateway

Deployment

​Config Structure

​Basic Config

​Complete Config Example

​Config Fields

​Provider Configuration

​Strategy Configuration

​Retry Configuration

​Cache Configuration

​Timeout Configuration

​Guardrails Configuration

​Provider-Specific Fields

​Azure OpenAI

​Google Vertex AI

​AWS Bedrock

​OpenAI

​Config Inheritance

​Config Validation

​Validation Rules

​Passing Configs

​Via Header

​Via SDK

​Via Environment

​Config Conversion

​Override Parameters

​Forward Headers

​OpenAI Compliance

​Best Practices

Version Your Configs

Use Virtual Keys

Test Configs Locally

Monitor Config Performance

​Next Steps

Routing

Guardrails

Caching

Load Balancing

Build docs developers (and LLMs) love

Config Structure

Basic Config

Complete Config Example

Config Fields

Provider Configuration

Strategy Configuration

Retry Configuration

Cache Configuration

Timeout Configuration

Guardrails Configuration

Provider-Specific Fields

Azure OpenAI

Google Vertex AI

AWS Bedrock

OpenAI

Config Inheritance

Config Validation

Validation Rules

Passing Configs

Via Header

Via SDK

Via Environment

Config Conversion

Override Parameters

Forward Headers

OpenAI Compliance

Best Practices

Next Steps