Skip to main content

​
Overview

Interview Simulator uses a provider pattern for AI integrations, allowing multiple LLM providers to work together with automatic fallback. The system supports Google Gemini and OpenRouter out of the box, with extensibility for custom providers.

​
Provider Architecture

​
Provider Interface

All AI providers implement the AIProvider protocol defined in client/ai_provider.py:
client/ai_provider.py
from typing import Protocol

class AIProvider(Protocol):
    def generate_text(self, prompt: str) -> str:
        ...
Any class implementing this generate_text method can be used as a provider.

​
Provider Manager

The ProviderManager class in client/ai_provider_manager.py orchestrates multiple providers with circuit breaker pattern:
client/ai_provider_manager.py
class ProviderManager:
    def __init__(self, providers: list[AIProvider]):
        self.providers = providers
        self.fail_count = {p: 0 for p in providers}
        self.open_until = {p: 0 for p in providers}

    def generate_text(self, prompt):
        last_error = None

        for provider in self.providers:
            if not self._is_available(provider):
                continue

            try:    
                return provider.generate_text(prompt)
            except Exception as e:
                last_error = e
                self.fail_count[provider] += 1

                if self.fail_count[provider] >= 3:
                    self.open_until[provider] = time.time() + 120.0

        raise AIServiceError(f"All providers failed: {last_error}")
Key features:
  • Automatic fallback: Tries providers in order until one succeeds
  • Circuit breaker: After 3 failures, a provider is disabled for 120 seconds
  • Resilience: System continues working if one provider fails

​
Initialization

Providers are initialized in app/extensions.py during application startup:
app/extensions.py
def init_ai_providers(app):
    global provider_manager, ai_client
    openrouter_key = app.config.get("OPENROUTER_API_KEY", "")
    gemini_key = app.config.get("GEMINI_API_KEY", "")

    providers = []

    if openrouter_key:
        providers.append(
            OpenRouterProvider(
                api_key=openrouter_key, model_name="openai/gpt-oss-20b:free"
            )
        )

    if gemini_key:
        providers.append(
            GeminiProvider(api_key=gemini_key, model_name="gemini-2.5-flash")
        )

    if not providers:
        app.logger.warning("No AI providers configured! Check your API keys.")
        return

    provider_manager = ProviderManager(providers)
    ai_client = AIClient(provider_manager)
    app.logger.info(f"Initialized {len(providers)} AI provider(s)")

​
Built-in Providers

​
Google Gemini

The Gemini provider uses Google’s Generative AI SDK:
client/gemini_provider.py
from google import genai
from tenacity import retry, stop_after_attempt, wait_exponential

class GeminiProvider:
    def __init__(self, api_key: str, model_name: str = "gemini-2.5-flash"):
        if not api_key:
            raise ValueError("API key is required")

        self.client = genai.Client(api_key=api_key)
        self.model_name = model_name

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=2, max=10),
        retry=retry_if_exception_type((ConnectionError, TimeoutError)),
    )
    def generate_text(self, prompt: str) -> str:
        response = self.client.models.generate_content(
            model=self.model_name,
            contents=prompt,
        )

        if not response.parts:
            raise RuntimeError("Gemini response was blocked or empty")

        text = response.text
        if not text:
            raise RuntimeError("Gemini returned empty response")

        return text
Configuration: 
GEMINI_API_KEY=your-api-key-here
Features:
  • Automatic retry with exponential backoff (3 attempts)
  • Response validation (checks for blocked or empty responses)
  • Configurable model selection
Get API key: https://ai.google.dev/

​
OpenRouter

The OpenRouter provider provides access to multiple LLM providers through a unified API:
client/openrouter_provider.py
import requests
import json
from tenacity import retry, stop_after_attempt, wait_exponential

class OpenRouterProvider:
    def __init__(self, api_key: str, model_name: str = "openai/gpt-oss-20b:free"):
        if not api_key:
            raise ValueError("API key is required")
        self.api_key = api_key
        self.model_name = model_name
        self.endpoint = "https://openrouter.ai/api/v1/chat/completions"
        self._headers = {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json",
        }

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=2, max=10),
        retry=retry_if_exception_type(
            (requests.RequestException, ConnectionError, TimeoutError)
        ),
    )
    def generate_text(self, prompt: str) -> str:
        payload = {
            "model": self.model_name,
            "messages": [{"role": "user", "content": prompt}],
            "extra_body": {"reasoning": {"enabled": True}},
        }

        response = requests.post(
            self.endpoint, headers=self._headers, data=json.dumps(payload)
        )

        if response.status_code != 200:
            raise RuntimeError(
                f"OpenRouter API error: {response.status_code} - {response.text}"
            )

        data = response.json()
        try:
            message = data["choices"][0]["message"]
            text = message.get("content")
        except (KeyError, IndexError):
            raise RuntimeError("Malformed response from OpenRouter API")

        if not text:
            raise RuntimeError("OpenRouter returned empty response")

        return text
Configuration: 
OPENROUTER_API_KEY=your-api-key-here
Features:
  • Automatic retry with exponential backoff (3 attempts)
  • Reasoning mode enabled by default
  • Response validation and error handling
  • Access to multiple LLM providers through one API
Get API key: https://openrouter.ai/ Available models: See OpenRouter models

​
Configuration Options

​
Selecting Providers

Configure which providers to use via environment variables:
.env
# Use both providers (default)
GEMINI_API_KEY=your-gemini-key
OPENROUTER_API_KEY=your-openrouter-key
ACTIVE_PROVIDERS=openrouter,gemini

# Use only Gemini
GEMINI_API_KEY=your-gemini-key
ACTIVE_PROVIDERS=gemini

# Use only OpenRouter
OPENROUTER_API_KEY=your-openrouter-key
ACTIVE_PROVIDERS=openrouter
At least one provider must be configured. The application will log a warning if no API keys are set.

​
Changing Models

To use different models, modify the initialization in app/extensions.py:
app/extensions.py
# Use Gemini Pro instead of Flash
if gemini_key:
    providers.append(
        GeminiProvider(api_key=gemini_key, model_name="gemini-2.5-pro")
    )

# Use a different OpenRouter model
if openrouter_key:
    providers.append(
        OpenRouterProvider(
            api_key=openrouter_key, 
            model_name="anthropic/claude-3.5-sonnet"
        )
    )

​
Provider Priority

Providers are tried in the order they’re added to the list. To change priority, reorder the initialization:
app/extensions.py
providers = []

# Try Gemini first
if gemini_key:
    providers.append(GeminiProvider(...))

# Then fall back to OpenRouter
if openrouter_key:
    providers.append(OpenRouterProvider(...))

​
Adding Custom Providers

Create a new provider by implementing the AIProvider protocol:
1

Create provider class

Create a new file client/custom_provider.py:
client/custom_provider.py
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type,
)

class CustomProvider:
    def __init__(self, api_key: str, model_name: str = "default-model"):
        if not api_key:
            raise ValueError("API key is required")
        self.api_key = api_key
        self.model_name = model_name
        # Initialize your client here

    @retry(
        stop=stop_after_attempt(3),
        wait=wait_exponential(multiplier=1, min=2, max=10),
        retry=retry_if_exception_type((ConnectionError, TimeoutError)),
    )
    def generate_text(self, prompt: str) -> str:
        # Implement your API call here
        # Must return a string
        response = self._call_api(prompt)
        return response

    def _call_api(self, prompt: str) -> str:
        # Your implementation
        pass
2

Add configuration

Add environment variable to app/config.py:
app/config.py
class Config:
    # ... existing config ...
    CUSTOM_API_KEY = os.getenv("CUSTOM_API_KEY", "")
Add to .env.example:
CUSTOM_API_KEY=
3

Register provider

Add initialization logic to app/extensions.py:
app/extensions.py
from client.custom_provider import CustomProvider

def init_ai_providers(app):
    # ... existing code ...
    custom_key = app.config.get("CUSTOM_API_KEY", "")

    if custom_key:
        providers.append(
            CustomProvider(
                api_key=custom_key,
                model_name="your-model-name"
            )
        )
4

Test provider

Set your API key and test:
CUSTOM_API_KEY=your-key-here
python wsgi.py
The application logs will show:
Initialized 3 AI provider(s)

​
Retry and Error Handling

All providers use the tenacity library for automatic retries: 
@retry(
    stop=stop_after_attempt(3),  # Maximum 3 attempts
    wait=wait_exponential(multiplier=1, min=2, max=10),  # Exponential backoff
    retry=retry_if_exception_type((ConnectionError, TimeoutError)),
)
def generate_text(self, prompt: str) -> str:
    # Implementation
Retry behavior:
  • 1st attempt: Immediate
  • 2nd attempt: Wait 2 seconds
  • 3rd attempt: Wait 4 seconds
After 3 failures, the ProviderManager’s circuit breaker opens for 120 seconds.

​
Circuit Breaker Pattern

The ProviderManager implements a circuit breaker to prevent cascading failures:
client/ai_provider_manager.py
def generate_text(self, prompt):
    for provider in self.providers:
        if not self._is_available(provider):  # Circuit open?
            continue

        try:    
            return provider.generate_text(prompt)
        except Exception as e:
            self.fail_count[provider] += 1

            if self.fail_count[provider] >= 3:
                # Open circuit for 120 seconds
                self.open_until[provider] = time.time() + 120.0
States:
  • Closed: Provider accepts requests (normal operation)
  • Open: Provider temporarily disabled after 3 consecutive failures
  • Half-open: After 120 seconds, provider accepts requests again

​
AI Client Layer

The AIClient class in client/ai_client.py provides high-level methods for interview features:
client/ai_client.py
class AIClient:
    def __init__(self, provider_manager):
        self.provider_manager = provider_manager

    def generate_first_question(self, cv_text, job_desc, job_title, company_name) -> str:
        prompt = PromptTemplates.first_question_generation(...)
        return self._generate(prompt)

    def generate_followup_question(self, convo_history, cv_text, job_desc, ...) -> str:
        prompt = PromptTemplates.followup_question_generation(...)
        return self._generate(prompt)

    def generate_feedback(self, convo_history, cv_text, job_desc, job_title) -> dict:
        prompt = PromptTemplates.feedback_generation(...)
        return self._parse_json(self._generate(prompt), expect_list=False)

    def _generate(self, prompt: str) -> str:
        try:
            return self.provider_manager.generate_text(prompt)
        except Exception as e:
            raise AIServiceError(f"AI generation failed: {e}")
The AIClient abstracts provider details from the rest of the application.

​
Best Practices

  1. Use multiple providers: Configure both Gemini and OpenRouter for redundancy
  2. Monitor failures: Check application logs for provider failures
  3. Validate responses: All providers validate response format before returning
  4. Configure retries: Use tenacity for automatic retry with backoff
  5. Handle empty responses: Check for empty/blocked responses before using
  6. Set appropriate models: Choose models based on cost/performance tradeoffs

​
Troubleshooting

​
”No AI providers configured” warning

Cause: No API keys set in .env file. Solution: Set at least one provider API key:
.env
GEMINI_API_KEY=your-key-here

​
“All providers failed” error

Cause: All configured providers are returning errors or have open circuit breakers. Solution:
  1. Check API key validity
  2. Verify network connectivity
  3. Check provider service status
  4. Review application logs for specific error messages
  5. Wait 120 seconds for circuit breakers to reset

​
Empty or blocked responses

Cause: Provider filtered the response due to safety/content policies. Solution:
  1. Review prompt content
  2. Try a different provider
  3. Adjust prompt templates to avoid policy violations

​
Rate limiting errors

Cause: Exceeded provider API rate limits. Solution:
  1. Implement request throttling
  2. Use multiple providers for load distribution
  3. Upgrade to higher-tier API plan
  4. Cache responses when possible

Build docs developers (and LLMs) love

Get started for freeTalk to us