Skip to main content

​
Overview

The analytics.getFailures endpoint provides detailed information about failed AI invocations, including workflow errors, tool call failures, and execution telemetry. This is essential for debugging AI behavior, identifying reliability issues, and improving model performance.

​
Endpoint

orpc.analytics.getFailures.queryOptions({ 
  input: { limit, variant } 
})

​
Input Parameters

​
limit
number
default:50
Maximum number of failure entries to return (1-100)
​
variant
enum
default:"all"
Filter failures by AI variant:
  • all - All variants
  • Apex - Apex variant only
  • Trendsurfer - Trendsurfer variant only
  • Contrarian - Contrarian variant only
  • Sovereign - Sovereign variant only

​
Response Schema

​
failures
array
Array of failure entries with detailed debugging information.
​
modelStats
array
Aggregated failure statistics for each model.

​
Failure Detection Logic

The system detects failures using multiple signals:

​
Workflow Failures

Detected when the AI response contains error indicators: 
const hasErrorInResponse =
  response.toLowerCase().includes('error:') ||
  response.toLowerCase().includes('error occurred') ||
  response.toLowerCase().includes('failed to') ||
  response.toLowerCase().includes('aborted') ||
  response.toLowerCase().includes('exception');

const failureReason = 
  responsePayload?.failureReason ?? 
  responsePayload?.error ?? 
  null;

const isWorkflowFailure = hasErrorInResponse || !!failureReason;
Common workflow failures:
  • AI model timeouts
  • Rate limiting errors
  • Invalid AI responses
  • Prompt validation failures
  • System exceptions

​
Tool Call Failures

Detected when tool call metadata indicates failure: 
const isToolCallFailure = toolCallMetadata.some((metadata) => {
  const parsed = JSON.parse(metadata);
  return parsed?.results?.some?.((r) => r.success === false);
});
Common tool call failures:
  • API endpoint errors (exchange, data provider)
  • Validation failures (invalid parameters)
  • Insufficient balance
  • Order rejection
  • Network timeouts

​
Excluded from Failures

The following are not counted as failures:
  • Pending invocations (“No response yet”, “Pending”, empty responses)
  • Intentional skips (“No trade opportunity detected”)
  • Valid “pass” decisions

​
Step Telemetry Debugging

Step telemetry provides detailed execution traces for debugging: 
interface StepTelemetry {
  stepNumber: number;         // Sequential step (1, 2, 3...)
  toolNames: string[];        // Tools called in this step
  inputTokens: number;        // Prompt tokens
  outputTokens: number;       // Completion tokens
  totalTokens: number;        // Sum of input + output
  timestamp: string;          // When this step executed
}
Use cases:
  • Token usage analysis: Identify expensive prompts or long completions
  • Execution flow: See which tools were called and in what order
  • Performance profiling: Measure time between steps
  • Failure point identification: Find the exact step where execution failed

​
Usage Examples

​
Display Recent Failures

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function RecentFailures() {
  const { data, isLoading } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 20, variant: 'all' }
    })
  );

  if (isLoading) return <div>Loading...</div>;

  return (
    <div>
      <h2>Recent Failures</h2>
      {data?.failures.map((failure) => (
        <div key={failure.invocationId} className="failure-card">
          <h3>{failure.modelName}</h3>
          <p><strong>ID:</strong> {failure.invocationId}</p>
          <p><strong>Time:</strong> {failure.createdAt.toLocaleString()}</p>
          
          {failure.failureReason && (
            <div className="error">
              <strong>Reason:</strong> {failure.failureReason}
            </div>
          )}
          
          <details>
            <summary>Response ({failure.response.length} chars)</summary>
            <pre>{failure.response}</pre>
          </details>
          
          <p><strong>Tool Calls:</strong> {failure.toolCalls.length}</p>
          {failure.totalSteps && (
            <p><strong>Steps:</strong> {failure.totalSteps}</p>
          )}
        </div>
      ))}
    </div>
  );
}

​
Model Failure Statistics Dashboard

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function FailureStatsDashboard() {
  const { data } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 50, variant: 'all' }
    })
  );

  return (
    <div>
      <h2>Model Reliability</h2>
      <table>
        <thead>
          <tr>
            <th>Model</th>
            <th>Variant</th>
            <th>Invocations</th>
            <th>Workflow Failures</th>
            <th>Tool Failures</th>
            <th>Failure Rate</th>
          </tr>
        </thead>
        <tbody>
          {data?.modelStats.map((stat) => (
            <tr key={stat.modelId}>
              <td>{stat.modelName}</td>
              <td>{stat.variant}</td>
              <td>{stat.invocationCount}</td>
              <td>{stat.failedWorkflowCount}</td>
              <td>{stat.failedToolCallCount}</td>
              <td className={stat.failureRate > 10 ? 'text-red' : 'text-green'}>
                {stat.failureRate.toFixed(2)}%
              </td>
            </tr>
          ))}
        </tbody>
      </table>
    </div>
  );
}

​
Step Telemetry Viewer

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function FailureTelemetryViewer({ invocationId }: { invocationId: string }) {
  const { data } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 100, variant: 'all' }
    })
  );

  const failure = data?.failures.find((f) => f.invocationId === invocationId);
  
  if (!failure) return <div>Failure not found</div>;

  return (
    <div>
      <h2>Execution Trace</h2>
      <p><strong>Model:</strong> {failure.modelName}</p>
      <p><strong>Total Steps:</strong> {failure.totalSteps || 'N/A'}</p>
      <p><strong>Total Tokens:</strong> {(failure.totalInputTokens || 0) + (failure.totalOutputTokens || 0)}</p>
      
      {failure.stepTelemetry && (
        <div>
          <h3>Step-by-Step Breakdown</h3>
          {failure.stepTelemetry.map((step) => (
            <div key={step.stepNumber} className="step-card">
              <h4>Step {step.stepNumber}</h4>
              <p><strong>Time:</strong> {new Date(step.timestamp).toLocaleTimeString()}</p>
              <p><strong>Tools:</strong> {step.toolNames.join(', ') || 'None'}</p>
              <p><strong>Tokens:</strong> {step.inputTokens} in, {step.outputTokens} out ({step.totalTokens} total)</p>
            </div>
          ))}
        </div>
      )}
      
      <h3>Tool Calls</h3>
      {failure.toolCalls.map((tc) => (
        <div key={tc.id}>
          <p><strong>{tc.toolCallType}</strong></p>
          <pre>{tc.metadata}</pre>
        </div>
      ))}
    </div>
  );
}

​
Variant-Specific Failure Analysis

import { useQuery } from '@tanstack/react-query';
import { orpc } from '@/server/orpc/client';

function ApexFailureAnalysis() {
  const { data } = useQuery(
    orpc.analytics.getFailures.queryOptions({
      input: { limit: 100, variant: 'Apex' }
    })
  );

  const failures = data?.failures || [];
  const stats = data?.modelStats || [];

  // Group failures by reason
  const failuresByReason = failures.reduce((acc, f) => {
    const reason = f.failureReason || 'Unknown';
    acc[reason] = (acc[reason] || 0) + 1;
    return acc;
  }, {} as Record<string, number>);

  return (
    <div>
      <h2>Apex Variant Failure Analysis</h2>
      
      <div>
        <h3>Overall Stats</h3>
        <p>Total Failures: {failures.length}</p>
        <p>Avg Failure Rate: {(stats.reduce((sum, s) => sum + s.failureRate, 0) / stats.length).toFixed(2)}%</p>
      </div>
      
      <div>
        <h3>Failure Reasons</h3>
        {Object.entries(failuresByReason).map(([reason, count]) => (
          <div key={reason}>
            <strong>{reason}:</strong> {count} occurrences
          </div>
        ))}
      </div>
      
      <div>
        <h3>Most Recent Failures</h3>
        {failures.slice(0, 5).map((f) => (
          <div key={f.invocationId}>
            <p>{f.modelName} - {f.createdAt.toLocaleString()}</p>
            <p>{f.failureReason || 'No reason provided'}</p>
          </div>
        ))}
      </div>
    </div>
  );
}

​
Source Code Reference

Implementation details:
  • Router: src/server/orpc/router/analytics.ts:228-241
  • Failure detection: src/server/features/analytics/queries.server.ts:383-442
  • Recent failures query: src/server/features/analytics/queries.server.ts:548-660
  • Failure stats query: src/server/features/analytics/queries.server.ts:447-543
  • Step telemetry normalization: src/server/features/analytics/queries.server.ts:70-108

Build docs developers (and LLMs) love

Get started for freeTalk to us