Multimodal Input

Gemini models can process multiple types of media in addition to text. This guide covers how to provide images, audio, video, and PDF files as input.

Images

There are three main ways to provide image input:

Cloud Storage (GCS)
Local Files (Bytes)
File API

Use Part.from_uri for images stored in Google Cloud Storage:

from google import genai
from google.genai import types

client = genai.Client(api_key='your-api-key')

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'What is this image about?',
        types.Part.from_uri(
            file_uri='gs://generativeai-downloads/images/scones.jpg',
            mime_type='image/jpeg',
        ),
    ],
)
print(response.text)

Supported image formats: JPEG, PNG, WebP, GIF

Use Part.from_bytes for local image files:

from google.genai import types

with open('your_image_path.jpg', 'rb') as f:
    image_bytes = f.read()

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'What is this image about?',
        types.Part.from_bytes(
            data=image_bytes,
            mime_type='image/jpeg'
        ),
    ],
)
print(response.text)

Best for images under 20MB. For larger files, use the File API.

Upload images to the File API first (Gemini Developer API only):

# Upload the image
file = client.files.upload(file='image.jpg')

# Use it in generate_content
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Describe this image', file]
)
print(response.text)

Best for large images or when reusing the same image multiple times.

Audio

Process audio files for transcription, analysis, or understanding:

Local Audio (Bytes)
Cloud Storage Audio
File API (Long Audio)

from google.genai import types

with open('audio_sample.mp3', 'rb') as f:
    audio_bytes = f.read()

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        types.Part.from_bytes(
            data=audio_bytes,
            mime_type='audio/mp3',
        ),
        'Transcribe this audio.'
    ]
)
print(response.text)

Supported audio formats: MP3, WAV, FLAC, AAC

from google.genai import types

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'What is being discussed in this audio?',
        types.Part.from_uri(
            file_uri='gs://your-bucket/audio.mp3',
            mime_type='audio/mp3',
        ),
    ],
)
print(response.text)

For long audio files, upload to the File API first:

# Upload
audio_file = client.files.upload(file='podcast.mp3')

# Wait for processing to complete
while audio_file.state == 'PROCESSING':
    time.sleep(2)
    audio_file = client.files.get(name=audio_file.name)

# Generate content
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[audio_file, 'Summarize this podcast.']
)
print(response.text)

Video

Analyze video content for descriptions, summaries, or specific questions:

Cloud Storage Video
File API (Recommended)
Video Frames

from google.genai import types

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'What happens in this video?',
        types.Part.from_uri(
            file_uri='gs://your-bucket/video.mp4',
            mime_type='video/mp4',
        ),
    ],
)
print(response.text)

Supported video formats: MP4, MOV, AVI, WebM, FLV, MPG

The File API is recommended for videos:

# Upload
video_file = client.files.upload(file='video.mp4')

# Wait for processing
import time
while video_file.state == 'PROCESSING':
    time.sleep(5)
    video_file = client.files.get(name=video_file.name)

# Generate content
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[video_file, 'What happens in this video?']
)
print(response.text)

Video files typically require processing time. Always check the file state before using it.

Extract and analyze specific frames:

from google.genai import types

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Describe what happens at the 30 second mark',
        types.Part.from_uri(
            file_uri='gs://your-bucket/video.mp4',
            mime_type='video/mp4',
        ),
    ],
)
print(response.text)

PDFs

Extract information from PDF documents:

File API (Gemini Developer API)
Cloud Storage (Vertex AI)
Multiple PDFs

# Upload PDF
pdf_file = client.files.upload(file='document.pdf')

# Wait for processing
import time
while pdf_file.state == 'PROCESSING':
    time.sleep(2)
    pdf_file = client.files.get(name=pdf_file.name)

# Ask questions about the PDF
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=['Summarize this document', pdf_file]
)
print(response.text)

from google.genai import types

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'What are the key findings in this research paper?',
        types.Part.from_uri(
            file_uri='gs://your-bucket/research-paper.pdf',
            mime_type='application/pdf',
        ),
    ],
)
print(response.text)

Analyze multiple PDFs together:

from google.genai import types

# Upload two PDFs
file1 = client.files.upload(file='paper1.pdf')
file2 = client.files.upload(file='paper2.pdf')

# Use them together
response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Compare and contrast these two research papers.',
        file1,
        file2
    ]
)
print(response.text)

Combining Multiple Modalities

You can mix different media types in a single request:

from google.genai import types

# Upload files
image_file = client.files.upload(file='chart.png')
audio_file = client.files.upload(file='presentation.mp3')

response = client.models.generate_content(
    model='gemini-2.5-flash',
    contents=[
        'Based on this chart and audio presentation, ',
        image_file,
        audio_file,
        'what are the main conclusions?'
    ]
)
print(response.text)

MIME Types Reference

Common MIME types for different media:

Media Type	MIME Type Examples
Images	`image/jpeg`, `image/png`, `image/webp`, `image/gif`
Audio	`audio/mp3`, `audio/wav`, `audio/flac`, `audio/aac`
Video	`video/mp4`, `video/mov`, `video/avi`, `video/webm`
PDF	`application/pdf`

File API Management

Manage uploaded files:

# Upload
file = client.files.upload(file='document.pdf')
print(f"Uploaded: {file.name}")

# Get file info
file_info = client.files.get(name=file.name)
print(f"State: {file_info.state}")
print(f"Size: {file_info.size_bytes} bytes")

# List all files
for f in client.files.list():
    print(f"{f.name}: {f.state}")

# Delete when done
client.files.delete(name=file.name)

Streaming with Multimodal Input

You can stream responses for multimodal inputs:

from google.genai import types

with open('image.jpg', 'rb') as f:
    image_bytes = f.read()

for chunk in client.models.generate_content_stream(
    model='gemini-2.5-flash',
    contents=[
        'Describe this image in detail',
        types.Part.from_bytes(data=image_bytes, mime_type='image/jpeg'),
    ],
):
    print(chunk.text, end='')

Use Cases

Document Analysis

Extract insights from PDFs, images of documents, and scanned files

Video Understanding

Analyze video content, generate descriptions, and answer questions

Audio Transcription

Transcribe and analyze audio content, podcasts, and meetings

Visual Q&A

Answer questions about images, charts, and diagrams

Best Practices

Use Part.from_uri for large files or files already in cloud storage
Use Part.from_bytes for small files (< 20MB) from local filesystem
Use the File API for files that need preprocessing (video, long audio, PDFs)
Always specify the correct MIME type for your media
Check file state (PROCESSING, ACTIVE) before using uploaded files
Delete files after use to manage storage costs
Combine multiple modalities when relevant to your use case
For Gemini Developer API, use the File API for all large files
For Vertex AI, you can use GCS URIs directly with Part.from_uri

Get Started

Core Concepts

Content Generation

Advanced Features

Media Generation

Files & Embeddings

Fine-tuning & Batch

Configuration

Images

Audio

Video

PDFs

Combining Multiple Modalities

MIME Types Reference

File API Management

Streaming with Multimodal Input

Use Cases

Document Analysis

Video Understanding

Audio Transcription

Visual Q&A

Best Practices

Build docs developers (and LLMs) love

Get Started

Core Concepts

Content Generation

Advanced Features

Media Generation

Files & Embeddings

Fine-tuning & Batch

Configuration

​Images

​Audio

​Video

​PDFs

​Combining Multiple Modalities

​MIME Types Reference

​File API Management

​Streaming with Multimodal Input

​Use Cases

Document Analysis

Video Understanding

Audio Transcription

Visual Q&A

​Best Practices

Build docs developers (and LLMs) love

Images

Audio

Video

PDFs

Combining Multiple Modalities

MIME Types Reference

File API Management

Streaming with Multimodal Input

Use Cases

Best Practices