Search methods overview
Global search
Map-reduce over community reportsBest for high-level, dataset-wide questions and thematic analysis.
Local search
Entity-centric retrievalBest for specific entity queries and detailed fact retrieval.
DRIFT search
Iterative graph traversalBest for complex multi-hop reasoning and exploratory analysis.
Quick comparison table
| Aspect | Global Search | Local Search | DRIFT Search |
|---|---|---|---|
| Question Type | Broad, thematic | Specific, factual | Complex, multi-hop |
| Data Source | Community reports | Entities + text + reports | Dynamic graph traversal |
| Context Building | Fixed (all reports at level) | Semantic retrieval | Iterative expansion |
| LLM Calls | Many (map-reduce) | Single | Multiple (iterative) |
| Cost | High | Low-Medium | Medium-High |
| Response Time | Slow (2-10s) | Fast (<2s) | Medium (2-5s) |
| Token Usage | High | Low-Medium | Medium-High |
| Coverage | Entire dataset | Focused | Adaptive |
| Depth | Summarized | Detailed | Multi-level |
When to use each method
Global search
- Ideal for
- Strengths
- Limitations
- Example
- “What are the main themes in this dataset?”
- “What are the top trends across all documents?”
- “Summarize the key findings”
- “What are the most significant events?”
- “How do different topics relate globally?”
Local search
- Ideal for
- Strengths
- Limitations
- Example
- “Who is Dr. Jordan Hayes?”
- “What is the relationship between X and Y?”
- “What are the properties of this entity?”
- “When did this specific event occur?”
- “What does the document say about X?”
DRIFT search
- Ideal for
- Strengths
- Limitations
- Example
- “How do organizations influence these events?”
- “What chain of events connects A to B?”
- “What patterns emerge from analyzing X, Y, and Z together?”
- “How are these entities indirectly connected?”
- “What are the multi-step dependencies?”
Side-by-side examples
Here’s how each method handles the same dataset but different query types:Dataset: “Operation Dulce” (sci-fi narrative)
Query 1: Broad thematic question
Query 1: Broad thematic question
Question: “What are the main themes in this story?”
- Global Search (Best)
- Local Search (Suboptimal)
- DRIFT Search (Moderate)
Response: Analyzes all community reports to identify overarching themes like government secrecy, alien contact, scientific ethics, and human cooperation.Why it works: Synthesizes information across the entire narrative structure.Cost: ~15,000 tokens
Query 2: Specific entity question
Query 2: Specific entity question
Question: “Who is Agent Mercer and what is their role?”
- Local Search (Best)
- Global Search (Suboptimal)
- DRIFT Search (Good)
Response: Retrieves Agent Mercer entity, related relationships, and relevant text chunks providing detailed information about their role, background, and actions.Why it works: Direct entity retrieval with supporting evidence.Cost: ~2,500 tokens
Query 3: Multi-hop reasoning
Query 3: Multi-hop reasoning
Question: “How do the government organizations indirectly influence the scientific research at the base?”
- DRIFT Search (Best)
- Local Search (Limited)
- Global Search (Moderate)
Response: Traces connections from government entities through intermediary actors to research activities, revealing multi-step influence patterns.Why it works: Designed for graph traversal and multi-hop reasoning.Cost: ~7,000 tokens
Query 4: Relationship query
Query 4: Relationship query
Question: “What is the relationship between Dr. Hayes and the Dulce facility?”
- Local Search (Best)
- Global Search (Limited)
- DRIFT Search (Good)
Response: Retrieves both entities and their connecting relationships with supporting text evidence.Why it works: Optimized for entity relationship queries.Cost: ~2,800 tokens
Decision flowchart
Is your question about a specific entity or relationship?
YES → Use Local SearchExamples:
- “Who is X?”
- “What is X’s relationship to Y?”
- “What are X’s properties?”
Does your question require understanding the entire dataset?
YES → Use Global SearchExamples:
- “What are the main themes?”
- “What are the key trends?”
- “Summarize this dataset”
Hybrid approaches
You can combine search methods for comprehensive analysis:Sequential querying
Validation strategy
Performance benchmarks
Based on typical “Operation Dulce” dataset queries:| Metric | Global | Local | DRIFT |
|---|---|---|---|
| Avg Response Time | 8.5s | 1.2s | 3.8s |
| Avg Tokens (Prompt) | 11,500 | 2,800 | 6,200 |
| Avg Tokens (Output) | 800 | 400 | 600 |
| Avg Cost (GPT-4) | $0.12 | $0.03 | $0.07 |
| Parallelizable | Yes | No | Partially |
Actual performance varies based on dataset size, query complexity, and configuration parameters.
Configuration comparison
- Global Search
- Local Search
- DRIFT Search
Cost optimization strategies
Use local search first
Start with low-cost local search; escalate only if needed
Adjust community level
Use level 1 for global search instead of level 2 to reduce tokens
Tune DRIFT conservatively
Keep
n_depth=2 and drift_k_followups=2 for most queriesUse summaries
Set
use_community_summary=True in global searchCommon pitfalls
Using global search for specific entities
Using global search for specific entities
Problem: Expensive and provides less detail than local search.Solution: Use local search for entity-specific queries.
Using local search for broad themes
Using local search for broad themes
Problem: May miss important information not connected to retrieved entities.Solution: Use global search for dataset-wide questions.
Over-parameterizing DRIFT
Over-parameterizing DRIFT
Problem: Setting
n_depth=5 and drift_k_followups=10 wastes tokens.Solution: Start with default parameters and increase only if needed.Not checking context data
Not checking context data
Problem: Accepting answers without verifying supporting evidence.Solution: Always inspect
result.context_data to see what was used.Choosing based on use case
- Research & analysis
- Q&A systems
- Investigation
- Content summarization
Primary: Global search for themes and patternsSecondary: Local search for validating specific claimsTertiary: DRIFT search for exploring connections between findings
Next steps
Global search
Deep dive into global search
Local search
Master local search techniques
DRIFT search
Learn DRIFT search methods
Query overview
Complete query documentation