Tracing - Mastra

Mastra’s tracing system captures complete execution paths through your agents and workflows, providing visibility into every step of processing.

What is Tracing?

Tracing creates a hierarchical record (trace) of all operations in a request:

Trace - Complete execution of a request (identified by traceId)
Span - Individual operation within a trace (agent call, tool execution, etc.)
Parent-Child Relationships - Spans nest to show execution flow

Basic Usage

Automatic Tracing

All agent and workflow operations are automatically traced:

import { Mastra } from '@mastra/core';

const mastra = new Mastra({
  observability: { enabled: true },
  agents: {
    myAgent: {
      name: 'My Agent',
      instructions: 'You are helpful',
      model: 'gpt-4',
    },
  },
});

// This creates a complete trace automatically
const result = await mastra.getAgent('myAgent').generate({
  messages: [{ role: 'user', content: 'What is 2+2?' }],
});

console.log('Trace ID:', result.traceId);
// Trace ID: a1b2c3d4e5f67890a1b2c3d4e5f67890

Retrieving Traces

Access trace data after execution:

const result = await agent.generate({
  messages: [{ role: 'user', content: 'Hello' }],
});

// Get the complete trace
const trace = await mastra.getTrace(result.traceId!);

console.log('Root span:', trace.rootSpan);
console.log('All spans:', trace.spans);
console.log('Span count:', trace.spans.length);

Span Types

Agent Run Span

Root span for agent execution:

const result = await agent.generate({
  messages: [{ role: 'user', content: 'Hello' }],
});

const trace = await mastra.getTrace(result.traceId!);
const agentSpan = trace.rootSpan;

console.log(agentSpan);
// {
//   type: 'agent_run',
//   name: 'myAgent',
//   entityType: 'agent',
//   entityId: 'myAgent',
//   attributes: {
//     conversationId: 'thread-123',
//     instructions: 'You are helpful',
//     availableTools: ['calculator'],
//     maxSteps: 5
//   },
//   startTime: Date(...),
//   endTime: Date(...),
//   input: [...messages],
//   output: [...response]
// }

Model Generation Span

Captures LLM API calls:

// Model generation span attributes
{
  type: 'model_generation',
  attributes: {
    model: 'gpt-4',
    provider: 'openai',
    
    // Token usage
    usage: {
      inputTokens: 150,
      outputTokens: 75,
      inputDetails: {
        text: 100,
        cacheRead: 50    // Anthropic cache hits
      },
      outputDetails: {
        text: 60,
        reasoning: 15     // o1 reasoning tokens
      }
    },
    
    // Model parameters
    parameters: {
      temperature: 0.7,
      maxOutputTokens: 1000,
      topP: 0.9
    },
    
    streaming: true,
    finishReason: 'stop',
    responseModel: 'gpt-4-0613',
    responseId: 'chatcmpl-123'
  }
}

Tool Call Span

Captures tool/function execution:

// Tool call span
{
  type: 'tool_call',
  name: 'calculator',
  attributes: {
    toolType: 'function',
    toolDescription: 'Perform calculations',
    success: true
  },
  input: { operation: 'add', a: 2, b: 2 },
  output: { result: 4 }
}

Workflow Spans

Captures workflow execution:

// Workflow run span
{
  type: 'workflow_run',
  name: 'myWorkflow',
  attributes: {
    status: 'completed'
  }
}

// Workflow step span
{
  type: 'workflow_step',
  name: 'processData',
  attributes: {
    status: 'completed'
  },
  input: { data: [...] },
  output: { processed: [...] }
}

Trace Metadata

Adding Metadata

Attach custom metadata to traces:

const result = await agent.generate({
  messages: [{ role: 'user', content: 'Hello' }],
  
  tracingOptions: {
    metadata: {
      userId: 'user-123',
      source: 'web-app',
      version: '1.0.0',
      experimentId: 'exp-456',
    },
  },
});

// Metadata is stored in root span
const trace = await mastra.getTrace(result.traceId!);
console.log(trace.rootSpan.metadata);
// { userId: 'user-123', source: 'web-app', ... }

Using Tags

Categorize traces with tags:

const result = await agent.generate({
  messages: [{ role: 'user', content: 'Hello' }],
  
  tracingOptions: {
    tags: ['production', 'user-query', 'priority-high'],
  },
});

// Query traces by tags (implementation depends on platform)
const traces = await observability.queryTraces({
  tags: ['production'],
});

Request Context Keys

Automatically extract fields from RequestContext:

import { RequestContext } from '@mastra/core/request-context';

const requestContext = new RequestContext();
requestContext.set('userId', 'user-123');
requestContext.set('session.experimentId', 'exp-456');
requestContext.set('user.tier', 'premium');

const result = await agent.generate({
  messages: [{ role: 'user', content: 'Hello' }],
  requestContext,
  
  tracingOptions: {
    // Extract these keys as metadata (supports dot notation)
    requestContextKeys: ['userId', 'session.experimentId', 'user.tier'],
  },
});

// Metadata includes extracted context
const trace = await mastra.getTrace(result.traceId!);
console.log(trace.rootSpan.metadata);
// { userId: 'user-123', session.experimentId: 'exp-456', user.tier: 'premium' }

Privacy Controls

Hiding Input/Output

Protect sensitive data:

const result = await agent.generate({
  messages: [{ role: 'user', content: 'My SSN is 123-45-6789' }],
  
  tracingOptions: {
    hideInput: true,   // Input won't be stored in spans
    hideOutput: true,  // Output won't be stored in spans
  },
});

Redacting Stream Data

Control what’s sent to clients in streams:

import { MastraServer } from '@mastra/hono';

const server = new MastraServer({
  app,
  mastra,
  streamOptions: {
    redact: true,  // Default: redacts system prompts, tool definitions, API keys
  },
});

Trace Linking

Parent Trace ID

Link traces across service boundaries:

// Service A creates a trace
const resultA = await agentA.generate({
  messages: [{ role: 'user', content: 'Start' }],
});

const traceId = resultA.traceId!;

// Service B continues the trace
const resultB = await agentB.generate({
  messages: [{ role: 'user', content: 'Continue' }],
  
  tracingOptions: {
    traceId,  // Continue existing trace
  },
});

// Both operations appear in same trace
console.log(resultA.traceId === resultB.traceId); // true

Parent Span ID

Create child spans in existing trace:

const resultA = await agentA.generate({
  messages: [{ role: 'user', content: 'Start' }],
});

const traceA = await mastra.getTrace(resultA.traceId!);
const parentSpanId = traceA.rootSpan.id;

// Create child span
const resultB = await agentB.generate({
  messages: [{ role: 'user', content: 'Continue' }],
  
  tracingOptions: {
    traceId: resultA.traceId!,
    parentSpanId,
  },
});

Annotations

Adding Scores

Add quality scores to traces:

const result = await agent.generate({
  messages: [{ role: 'user', content: 'Hello' }],
});

const trace = await mastra.getTrace(result.traceId!);

// Add score to trace
trace.addScore({
  name: 'accuracy',
  value: 0.95,
  comment: 'High accuracy response',
});

// Add score to specific span
const toolSpan = trace.spans.find(s => s.type === 'tool_call');
toolSpan?.addScore({
  name: 'relevance',
  value: 0.8,
});

Adding Feedback

Capture user feedback:

const trace = await mastra.getTrace(traceId);

trace.addFeedback({
  value: 1,  // thumbs up
  comment: 'Great response!',
});

// Or thumbs down
trace.addFeedback({
  value: -1,
  comment: 'Incorrect information',
});

Tree Traversal

Navigate span hierarchy:

const trace = await mastra.getTrace(traceId);

// Root span
const root = trace.rootSpan;
console.log(root.name);

// Child spans
root.children.forEach(child => {
  console.log(`  ${child.name}`);
  
  // Grandchildren
  child.children.forEach(grandchild => {
    console.log(`    ${grandchild.name}`);
  });
});

Finding Spans

Locate specific spans:

const trace = await mastra.getTrace(traceId);

// Find all tool call spans
const toolSpans = trace.spans.filter(s => s.type === 'tool_call');

// Find specific span by ID
const span = trace.getSpan('span-id-123');

// Find first model generation
const modelSpan = trace.spans.find(s => s.type === 'model_generation');

Performance Metrics

Calculate performance metrics from spans:

const trace = await mastra.getTrace(traceId);

// Total duration
const duration = trace.rootSpan.endTime! - trace.rootSpan.startTime;
console.log(`Total duration: ${duration}ms`);

// Token usage across all model calls
const modelSpans = trace.spans.filter(s => s.type === 'model_generation');
const totalTokens = modelSpans.reduce((sum, span) => {
  const usage = span.attributes?.usage;
  return sum + (usage?.inputTokens || 0) + (usage?.outputTokens || 0);
}, 0);
console.log(`Total tokens: ${totalTokens}`);

Get Started

Core Concepts

Agents

Workflows

Memory

RAG

Tools & MCP

Storage

Server & API

Observability

Evals

Deployment

​What is Tracing?

​Basic Usage

​Automatic Tracing

​Retrieving Traces

​Span Types

​Agent Run Span

​Model Generation Span

​Tool Call Span

​Workflow Spans

​Trace Metadata

​Adding Metadata

​Using Tags

​Request Context Keys

​Privacy Controls

​Hiding Input/Output

​Redacting Stream Data

​Trace Linking

​Parent Trace ID

​Parent Span ID

​Annotations

​Adding Scores

​Adding Feedback

​Span Navigation

​Tree Traversal

​Finding Spans

​Performance Metrics

​Next Steps

Logging