Tracing & Observability

​

Overview

• Request tracking - Trace every agent interaction
• Performance monitoring - Measure latency and bottlenecks
• Debugging - Inspect agent decisions and tool calls
• Analytics - Understand usage patterns and model behavior

​

Configuration

​

Environment Variables

TRACING=true

PHOENIX_API_KEY=your_api_key_here

PHOENIX_ENDPOINT=https://app.phoenix.arize.com/s/your_workspace/v1/traces

PHOENIX_PROJECT_NAME=my-junkie-bot

​

Setup Process

​

1. Installation

pip install arize-phoenix

​

2. Enable Tracing

TRACING=true
PHOENIX_API_KEY=your_phoenix_api_key
PHOENIX_PROJECT_NAME=junkie-production

​

3. Initialization

from core.observability import setup_phoenix_tracing

setup_phoenix_tracing()

​

Implementation Details

​

Lazy Initialization

def setup_phoenix_tracing():
    """Lazy initialization of Phoenix tracing with optimizations."""
    global _phoenix_tracer
    
    if _phoenix_tracer is not None:
        return _phoenix_tracer  # Already initialized
    
    if not TRACING_ENABLED:
        return None
    
    try:
        from phoenix.otel import register
        # ... initialization code
    except ImportError:
        logger.warning("Phoenix package not installed")
        return None

​

Environment Setup

if "PHOENIX_CLIENT_HEADERS" not in os.environ:
    os.environ["PHOENIX_CLIENT_HEADERS"] = f"api_key={PHOENIX_API_KEY}"

if "PHOENIX_COLLECTOR_ENDPOINT" not in os.environ:
    os.environ["PHOENIX_COLLECTOR_ENDPOINT"] = "https://app.phoenix.arize.com"

​

Tracer Registration

tracer_provider = register(
    project_name=PHOENIX_PROJECT_NAME,
    endpoint=PHOENIX_ENDPOINT,
    auto_instrument=True,  # Automatically trace LLM calls
    batch=True,            # Batch traces for better performance
)

• project_name - Organizes traces by project in Phoenix UI
• endpoint - Where traces are sent
• auto_instrument=True - Automatically captures LLM API calls
• batch=True - Reduces overhead by batching trace exports

​

Error Handling

​

Missing API Key

if not PHOENIX_API_KEY:
    logger.warning("PHOENIX_API_KEY not set, skipping Phoenix tracing")
    _phoenix_tracer = False
    return None

​

Missing Package

except ImportError:
    logger.warning("Phoenix tracing requested but 'phoenix' package not installed")
    _phoenix_tracer = False
    return None

​

Initialization Failure

except Exception as e:
    logger.error(f"Failed to initialize Phoenix tracing: {e}", exc_info=True)
    _phoenix_tracer = False
    return None

​

Performance Optimization

​

Batching

register(
    # ...
    batch=True,  # Batch traces for better performance
)

​

Conditional Import

if not TRACING_ENABLED:
    return None

try:
    from phoenix.otel import register  # Import only when needed
    # ...

​

Singleton Pattern

_phoenix_tracer = None

def setup_phoenix_tracing():
    global _phoenix_tracer
    
    if _phoenix_tracer is not None:
        return _phoenix_tracer  # Return cached instance

​

What Gets Traced

• LLM API calls - All requests to OpenAI-compatible APIs
• Model responses - Completions, token usage, latency
• Tool calls - Agent tool invocations and results
• Errors - Exceptions and failures in agent operations

​

Viewing Traces

1. Navigate to your Phoenix endpoint
2. Select your project (configured via PHOENIX_PROJECT_NAME)
3. View traces organized by:
   • Timestamp
   • Agent/model
   • Latency
   • Token usage
   • Success/error status

​

Example Configurations

TRACING=false

​

Troubleshooting

​

Tracing Not Working

1. Environment variable is set correctly:
   TRACING=true  # Not "True" or "TRUE"
   
2. API key is configured:
   PHOENIX_API_KEY=your_key_here
   
3. Phoenix package is installed:
   pip install arize-phoenix
   
4. Check logs for warnings: Look for messages like:
   • "PHOENIX_API_KEY not set, skipping Phoenix tracing"
   • "Phoenix package not installed"
   • "Failed to initialize Phoenix tracing"

​

High Overhead

1. Ensure batch=True is enabled (it is by default)
2. Consider using a sampling strategy (not currently implemented)
3. Disable tracing for high-frequency operations

​

Best Practices

1. Use different projects for different environments:
   # Production
   PHOENIX_PROJECT_NAME=junkie-prod
   
   # Development
   PHOENIX_PROJECT_NAME=junkie-dev
   
2. Enable tracing in development for debugging
3. Monitor token usage through traces to optimize costs
4. Review error traces to identify and fix issues
5. Keep tracing disabled in local development unless debugging