Multimodal

User: [Uploads screenshot of error message]
What's causing this error and how can I fix it?

Assistant: Looking at your screenshot, the error is a 
"TypeError: Cannot read property 'map' of undefined". This 
occurs on line 42 where you're trying to map over `data.items`.

The issue is that `data.items` is undefined, likely because:
1. The API response doesn't include an `items` property
2. The data hasn't loaded yet when the component renders

Fix:
```javascript
{data?.items?.map(item => ...)}
// or
{(data.items || []).map(item => ...)}

### Document Analysis

Analyze PDFs and documents:

### Chart and Graph Reading

Extract data from visualizations:

### OCR and Text Extraction

Read text from images:

## Agent Integration

Use multimodal capabilities with agents:

```typescript
{
  name: 'Visual Assistant',
  provider: 'openAI',
  model: 'gpt-4o',
  instructions: `
    You are a visual analysis assistant. When users upload images:
    1. Provide detailed descriptions
    2. Extract text and data
    3. Answer questions about the image
    4. Identify objects, people, and scenes
  `,
  tools: ['execute_code'],  // Can use code to process images
  tool_resources: {
    execute_code: {
      file_ids: []  // Images are available to code interpreter
    }
  }
}

from PIL import Image
import numpy as np
import matplotlib.pyplot as plt

# Load the uploaded image
img = Image.open('photo.jpg')
img_array = np.array(img)

# Analyze dimensions
height, width, channels = img_array.shape
print(f"Image size: {width}x{height}, Channels: {channels}")

# Calculate average color
avg_color = img_array.mean(axis=(0, 1))
print(f"Average RGB: {avg_color}")

# Create histogram
plt.figure(figsize=(10, 4))
for i, color in enumerate(['red', 'green', 'blue']):
    plt.subplot(1, 3, i+1)
    plt.hist(img_array[:,:,i].ravel(), bins=256, color=color, alpha=0.7)
    plt.title(f'{color.capitalize()} Channel')
plt.tight_layout()
plt.savefig('color_histogram.png')

{
  endpoint: 'openAI',
  model: 'gpt-4o',
  // Images are automatically processed
  // No special configuration needed
}

{
  endpoint: 'anthropic',
  model: 'claude-4.5-sonnet',
  // Supports PDFs natively with Responses API
}

{
  endpoint: 'google',
  model: 'gemini-2.0-flash-exp',
  // Multimodal by default
}

// Providers with document support
const documentSupportedProviders = new Set([
  'anthropic',
  'openAI',
  'bedrock',
  'google',
  'custom',  // Depends on endpoint
]);

User: [Uploads research_paper.pdf - 50 pages]
Summarize the methodology section.

Assistant: The methodology section (pages 8-15) describes:

**Study Design:**
- Randomized controlled trial
- 500 participants across 3 groups
- 12-month duration

**Data Collection:**
- Baseline assessments
- Monthly follow-ups
- Final evaluation at 12 months

**Analysis Methods:**
- Mixed-effects models for longitudinal data
- Intent-to-treat analysis
- Multiple imputation for missing data

[Detailed breakdown continues...]