NemoGuardrails

NemoGuardrails provides a framework for adding programmable constraints to LLM applications, ensuring safe, reliable, and compliant AI interactions. It enables you to control dialog flows, validate inputs and outputs, and implement safety policies.

Overview

NemoGuardrails enables you to:

Define dialog flows and conversation patterns
Validate and filter user inputs
Post-process and moderate model outputs
Implement fact-checking and hallucination detection
Enforce topical boundaries and content policies
Integrate with external validation services

When to Use NemoGuardrails

Content Safety

Prevent harmful, toxic, or inappropriate content in inputs and outputs

Compliance

Ensure AI responses meet regulatory requirements (GDPR, HIPAA, etc.)

Brand Safety

Maintain brand voice and prevent reputational risks

Domain Control

Keep conversations on-topic and within defined domains

Architecture

NemoGuardrails acts as a proxy between users and NIM:

Configuration

Complete Example

apiVersion: apps.nvidia.com/v1alpha1
kind: NemoGuardrail
metadata:
  name: nemoguardrails-sample
  namespace: nemo
spec:
  # Container image configuration
  image:
    repository: nvcr.io/nvidia/nemo-microservices/guardrails
    tag: "25.10"
    pullPolicy: IfNotPresent
    pullSecrets:
      - ngc-secret
  
  # Service exposure
  expose:
    service:
      type: ClusterIP
      port: 8000
  
  # Replica configuration
  replicas: 1
  
  # NIM endpoint for LLM queries
  nimEndpoint:
    baseURL: "http://meta-llama-3-1-8b-instruct.nemo.svc.cluster.local:8000/v1"
    # Optional: API key for hosted NIM endpoints
    # apiKeySecret: nim-api-key
    # apiKeyKey: NIM_ENDPOINT_API_KEY
  
  # Guardrail configuration storage
  configStore:
    pvc:
      name: "pvc-guardrail-config"
      create: true
      storageClass: ""
      volumeAccessMode: ReadWriteOnce
      size: "1Gi"
      # subPath: "guardrails"  # Optional subdirectory
    # Alternative: Use ConfigMap for configuration
    # configMap:
    #   name: guardrails-config
  
  # PostgreSQL database for conversation history
  databaseConfig:
    host: guardrail-pg-postgresql.nemo.svc.cluster.local
    port: 5432
    databaseName: guardraildb
    credentials:
      user: guardrailuser
      secretName: guardrail-pg-existing-secret
      passwordKey: password
  
  # OpenTelemetry tracing
  otel:
    enabled: true
    exporterOtlpEndpoint: http://guardrail-otel-collector.nemo.svc.cluster.local:4317
  
  # Resource limits
  resources:
    limits:
      cpu: "1"
      memory: "2Gi"
      ephemeral-storage: 10Gi
    requests:
      cpu: "500m"
      memory: "1Gi"
  
  # Monitoring
  metrics:
    enabled: true
    serviceMonitor:
      interval: 30s

Key Configuration Fields

spec.nimEndpoint

object

required

Configuration for the NIM endpoint that processes LLM requests.

baseURL

string

required

Full URL to NIM service including /v1 path. Must be a valid HTTP/HTTPS URL.

apiKeySecret

string

Name of Kubernetes secret containing NIM API key (for hosted NIMs).

apiKeyKey

string

default:"NIM_ENDPOINT_API_KEY"

Key in the secret that contains the API key value.

spec.configStore

object

required

Storage for guardrail configuration files. Must specify either PVC or ConfigMap.

pvc

object

Use persistent volume for configuration (recommended for production).

configMap

object

Use ConfigMap for configuration (suitable for simple setups).

spec.databaseConfig

object

PostgreSQL configuration for storing conversation history and context.

Guardrail Configuration

Configuration Structure

Guardrail configurations are stored in the configStore and follow the NeMo Guardrails format:

/config-store/
├── config.yml          # Main configuration
├── rails.co            # Colang rail definitions
├── prompts.yml         # Custom prompts
└── actions.py          # Custom actions (optional)

Example Configuration

Create ConfigMap or PVC

Using ConfigMap
Using PVC

apiVersion: v1
kind: ConfigMap
metadata:
  name: guardrails-config
  namespace: nemo
data:
  config.yml: |
    models:
      - type: main
        engine: nim
        model: meta/llama-3.1-8b-instruct
    
    rails:
      input:
        flows:
          - check jailbreak
          - check toxic language
      output:
        flows:
          - check hallucination
          - check factual accuracy
  
  rails.co: |
    define user ask about politics
      "What do you think about..."
      "Tell me about the election"
    
    define bot refuse politics
      "I'm not able to discuss political topics."
    
    define flow
      user ask about politics
      bot refuse politics

# Create PVC
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-guardrail-config
  namespace: nemo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
EOF

# Mount and populate configuration
kubectl run -it --rm config-loader \
  --image=busybox \
  --overrides='{
    "spec": {
      "volumes": [{
        "name": "config",
        "persistentVolumeClaim": {"claimName": "pvc-guardrail-config"}
      }],
      "containers": [{
        "name": "config-loader",
        "image": "busybox",
        "volumeMounts": [{"name": "config", "mountPath": "/config"}]
      }]
    }
  }' \
  -n nemo -- sh

# Inside the pod, create configuration files
cat > /config/config.yml <<EOF
models:
  - type: main
    engine: nim
    model: meta/llama-3.1-8b-instruct
EOF

Deploy NemoGuardrails

spec:
  configStore:
    configMap:
      name: guardrails-config
    # OR
    pvc:
      name: pvc-guardrail-config

Common Guardrail Patterns

define user ask about restricted topic
  "Tell me how to..."
  "Explain how to make..."

define bot refuse restricted topic
  "I cannot provide information on that topic."

define flow
  user ask about restricted topic
  bot refuse restricted topic
  bot offer alternative

Integration with NIM

Connecting to NIM

Guardrails acts as a transparent proxy to NIM:

nimEndpoint:
  baseURL: "http://meta-llama-3-1-8b-instruct.nemo.svc.cluster.local:8000/v1"

Using with NVIDIA Hosted NIM

For NVIDIA-hosted NIMs:

Create API Key Secret

kubectl create secret generic nim-api-key \
  --from-literal=NIM_ENDPOINT_API_KEY=<your-api-key> \
  -n nemo

Configure NIM Endpoint

nimEndpoint:
  baseURL: "https://integrate.api.nvidia.com/v1"
  apiKeySecret: nim-api-key
  apiKeyKey: NIM_ENDPOINT_API_KEY

API Usage

Chat Completion with Guardrails

curl -X POST http://nemoguardrails-sample.nemo.svc.cluster.local:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ],
    "config_id": "default"
  }'

Check Guardrail Status

curl http://nemoguardrails-sample.nemo.svc.cluster.local:8000/v1/health

List Available Configurations

curl http://nemoguardrails-sample.nemo.svc.cluster.local:8000/v1/configs

Database Configuration

PostgreSQL stores conversation history for context-aware guardrails:

databaseConfig:
  host: guardrail-pg-postgresql.nemo.svc.cluster.local
  port: 5432
  databaseName: guardraildb
  credentials:
    user: guardrailuser
    secretName: guardrail-pg-existing-secret
    passwordKey: password

Database is optional but recommended for production deployments with conversation memory.

Best Practices

Configuration Management

Version control your guardrail configurations
Use PVC for production (easier updates without pod restarts)
Test configurations in development before deploying
Document your rail definitions clearly

Performance Optimization

Run multiple replicas for high availability
Use efficient rail definitions (avoid complex regex)
Cache frequently used validations
Monitor latency added by guardrails

Security

Implement input sanitization rails
Validate all user inputs before LLM processing
Filter sensitive information from outputs
Rotate database credentials regularly

Monitoring

Enable OpenTelemetry for request tracing
Track guardrail activation rates
Monitor false positive/negative rates
Alert on configuration errors

Advanced Features

Custom Actions

Implement custom Python actions for complex logic:

# actions.py
from typing import Optional

async def check_company_policy(context: dict) -> bool:
    """Custom action to validate against company policies."""
    user_query = context.get("user_message")
    
    # Call external API or database
    response = await external_policy_checker(user_query)
    
    return response["compliant"]

def init(app):
    """Register custom actions."""
    app.register_action(check_company_policy)

Multi-Config Support

Serve multiple guardrail configurations:

/config-store/
├── default/
│   ├── config.yml
│   └── rails.co
├── strict/
│   ├── config.yml
│   └── rails.co
└── permissive/
    ├── config.yml
    └── rails.co

Select config in API calls:

curl -X POST .../v1/chat/completions \
  -d '{"config_id": "strict", ...}'

Troubleshooting

Guardrails Not Applied

Check:

ConfigMap/PVC is correctly mounted
Configuration syntax is valid (use YAML linter)
Rail definitions match expected patterns
Check logs for configuration errors

High Latency

Solutions:

Optimize rail complexity
Increase replica count
Use caching for repeated validations
Profile slow actions/validators

Database Connection Errors

Verify:

PostgreSQL is running and accessible
Credentials are correct
Database exists and has proper schema
Network policies allow connection

Next Steps

NIM Integration

Deploy NIM with guardrails

Colang Reference

Learn Colang syntax

Custom Actions

Build custom validators

Examples

Explore guardrail patterns

Get Started

Core Concepts

NIM Services

NeMo Microservices

Configuration

Operations

Overview

When to Use NemoGuardrails

Content Safety

Compliance

Brand Safety

Domain Control

Architecture

Configuration

Complete Example

Key Configuration Fields

Guardrail Configuration

Configuration Structure

Example Configuration

Common Guardrail Patterns

Integration with NIM

Connecting to NIM

Using with NVIDIA Hosted NIM

API Usage

Chat Completion with Guardrails

Check Guardrail Status

List Available Configurations

Database Configuration

Best Practices

Advanced Features

Custom Actions

Multi-Config Support

Troubleshooting

Next Steps

NIM Integration

Colang Reference

Custom Actions

Examples

Build docs developers (and LLMs) love

Get Started

Core Concepts

NIM Services

NeMo Microservices

Configuration

Operations

​Overview

​When to Use NemoGuardrails

Content Safety

Compliance

Brand Safety

Domain Control

​Architecture

​Configuration

​Complete Example

​Key Configuration Fields

​Guardrail Configuration

​Configuration Structure

​Example Configuration

​Common Guardrail Patterns

​Integration with NIM

​Connecting to NIM

​Using with NVIDIA Hosted NIM

​API Usage

​Chat Completion with Guardrails

​Check Guardrail Status

​List Available Configurations

​Database Configuration

​Best Practices

​Advanced Features

​Custom Actions

​Multi-Config Support

​Troubleshooting

​Next Steps

NIM Integration

Colang Reference

Custom Actions

Examples

Build docs developers (and LLMs) love

Overview

When to Use NemoGuardrails

Architecture

Configuration

Complete Example

Key Configuration Fields

Guardrail Configuration

Configuration Structure

Example Configuration

Common Guardrail Patterns

Integration with NIM

Connecting to NIM

Using with NVIDIA Hosted NIM

API Usage

Chat Completion with Guardrails

Check Guardrail Status

List Available Configurations

Database Configuration

Best Practices

Advanced Features

Custom Actions

Multi-Config Support

Troubleshooting

Next Steps