Testing

​

Overview

• ctester - Provider compatibility testing for LLM providers
• etester - Embedding provider and vector database testing
• ftester - Function and tool testing for agent capabilities

​

Testing Tools

​

ctester - Compatibility Tester

​

Features

• Tests multiple LLM providers (OpenAI, Anthropic, Gemini, Bedrock, Ollama, Custom)
• Evaluates agent-specific models for different roles
• Supports parallel testing with configurable workers
• Tests basic and advanced capabilities
• Validates JSON output, streaming, and reasoning features
• Generates comprehensive test reports

​

Usage

# Test a provider with default settings
./ctester -type openai -env .env

# Test custom provider with config file
./ctester \
  -type custom \
  -config /path/to/provider.yml \
  -env .env

​

Command-Line Options

​

Supported Agent Types

• simple - Simple conversation agent
• simple_json - JSON-structured response agent
• primary_agent - Main orchestration agent
• assistant - User-facing assistant agent
• generator - Content generation agent
• refiner - Response refinement agent
• adviser - Strategic advice agent
• reflector - Self-reflection agent
• searcher - Information search agent
• enricher - Context enrichment agent
• coder - Code generation agent
• installer - Tool installation agent
• pentester - Penetration testing agent

​

Test Groups

1. Basic - Fundamental capabilities (text generation, reasoning)
2. Advanced - Complex multi-step tasks
3. JSON - Structured output validation
4. Knowledge - Domain-specific understanding

​

Example Output

Testing openai Provider
=================================================
Agent: primary_agent | Model: gpt-4.1-mini
  Basic Tests:
    ✓ simple_generation (1250ms)
    ✓ reasoning_task (2100ms)
    ✓ context_understanding (1800ms)
  Advanced Tests:
    ✓ multi_step_reasoning (3500ms)
    ✓ tool_usage (2200ms)
  Total: 5/5 passed | Avg: 2170ms | Reasoning: Yes

Agent: coder | Model: gpt-4.1
  Basic Tests:
    ✓ code_generation (1900ms)
    ✓ code_explanation (1600ms)
  Advanced Tests:
    ✓ code_refactoring (2800ms)
    ✓ vulnerability_analysis (3200ms)
  Total: 4/4 passed | Avg: 2375ms | Reasoning: Yes

=================================================
Summary: 9/9 tests passed across 2 agents

​

etester - Embedding Tester

​

Features

• Tests embedding provider connectivity
• Validates pgvector extension setup
• Database statistics and health checks
• Document search and similarity testing
• Batch reindexing capabilities
• Database flush operations

​

Usage

# Test embedding provider and database
./etester test -verbose

# With custom environment file
./etester -env .env.prod test

​

Commands

​

Command-Line Options

​

Example Output

Embedding Tester (etester)
==========================
✓ Environment loaded
✓ Database connection established
✓ Embedding provider initialized

Testing embedding generation...
✓ Generated embedding: 1536 dimensions
✓ Embedding time: 245ms

Testing vector search...
✓ Inserted test document
✓ Similarity search: 3 results found
✓ Top match similarity: 0.9234

All tests passed!

​

ftester - Function Tester

​

Features

• Tests agent tool functions
• Validates Docker integration
• Tests database operations
• Supports mock mode for development
• Provider-specific testing
• Integration with observability stack

​

Usage

# Test with mock mode (no real flow)
./ftester -provider openai

# Test with real flow ID
./ftester -flow 123 -user 1 -provider anthropic

​

Command-Line Options

​

Example Output

Function Tester (ftester)
========================
Environment file: .env
Provider: openai
Flow ID: 123
------------------------

✓ Database connected
✓ Docker client initialized
✓ Provider controller ready
✓ Testing worker initialized

Executing function tests...
✓ Tool execution: run_command
✓ Container management: create_worker
✓ Database operations: store_result
✓ Provider interaction: generate_response

All function tests passed!

​

Running Unit Tests

​

Backend Unit Tests

# From backend directory
cd backend

# Run all tests
go test ./...

# With verbose output
go test -v ./...

# With coverage
go test -cover ./...

​

Frontend Unit Tests

# From frontend directory
cd frontend

# Run tests once
npm test

# Run in watch mode
npm run test:watch

# With coverage
npm run test:coverage

​

Integration Testing

# Start test environment
docker compose -f docker-compose.yml up -d

# Wait for services to be ready
sleep 30

# Run ctester for provider compatibility
docker exec pentagi /opt/pentagi/bin/ctester \
  -type openai \
  -groups basic \
  -verbose

# Run etester for embeddings
docker exec pentagi /opt/pentagi/bin/etester test

# Cleanup
docker compose down

​

Continuous Integration

1. Lint & Format Check
   • Backend: golangci-lint
   • Frontend: eslint and prettier
2. Unit Tests
   • Backend: go test
   • Frontend: npm test
3. Build Verification
   • Backend binaries
   • Frontend production build
   • Docker image build
4. Integration Tests
   • Provider compatibility (ctester)
   • Embedding tests (etester)
   • Function tests (ftester)

​

Test Best Practices

​

Troubleshooting Tests

# Increase timeout for slow tests
go test -timeout 5m ./...

# Or skip long tests
go test -short ./...

# Ensure PostgreSQL is running
docker ps | grep postgres

# Check DATABASE_URL in .env
cat .env | grep DATABASE_URL

# Test connection
psql $DATABASE_URL -c "SELECT 1;"

# Check API keys are set
env | grep -E '(OPEN_AI|ANTHROPIC|GEMINI)'

# Test with verbose mode
./ctester -type openai -verbose

# Check Docker is running
docker info

# Fix permissions
sudo chmod 666 /var/run/docker.sock

# Or add user to docker group
sudo usermod -aG docker $USER

​

Next Steps

• Learn about contributing
• Understand the architecture
• Configure LLM providers
• Set up observability