Overview
TheSearchEngine class handles all search operations for the RAG system, including vector search, keyword search, and hybrid search functionality.
Class Definition
SearchEngine
Constructor
__init__(config, database_manager)
Initialize the search engine with configuration and database access.
Configuration object containing search settings
Database manager for executing queries
- Loading the embedding model (SentenceTransformer)
- Creating a thread pool executor for CPU-bound embedding operations
- Configuring hybrid search parameters
Core Methods
search_similar_chunks(query, max_results=None, target_quarter=None)
Search for similar chunks using hybrid vector + keyword search with optional quarter filtering.Search query text
Maximum number of results to return. If None, uses chunks_per_quarter from config (default: 15)
Target quarter for filtering (e.g., “2025_q1”). Use “multiple” or None for no quarter filter
List[Dict[str, Any]] - List of chunk dictionaries with similarity scores
This is a synchronous method. For async contexts, use the async version of search methods.
encode_query_async(query)
Generate query embedding asynchronously using thread pool executor.Query text to encode
np.ndarray - Query embedding vector
This method prevents blocking the event loop while encoding queries, making it suitable for async contexts.
Advanced Search Methods
search_with_queries_async(queries, target_quarters, target_quarter, ticker=None)
Run multiple queries in parallel, merge and deduplicate by citation.List of search queries to execute in parallel
List of quarters to search (e.g., [‘2025_q1’, ‘2025_q2’])
Single quarter identifier or “multiple”
Optional ticker symbol to filter results
List[Dict[str, Any]] - Deduplicated and merged results
follow_up_search_async(question, has_tickers, is_general_question, is_multi_ticker, tickers_to_process, target_quarter, target_quarters)
Perform a single follow-up search with hybrid search (vector + keyword).Follow-up question to search
Whether the query includes ticker symbols
Whether this is a general (non-ticker-specific) question
Whether multiple tickers are involved
List of ticker symbols to process
Target quarter or None
List of quarters to search
List[Dict[str, Any]] - Search results
follow_up_search_parallel_async(follow_up_questions, …)
Run multiple follow-up questions in parallel, merge and deduplicate by citation.List of follow-up questions to execute in parallel
follow_up_search_async
Returns: List[Dict[str, Any]] - Deduplicated merged results
Internal Methods
_search_multiple_quarters_async(query, target_quarters, chunks_per_quarter=None)
Async version of multi-quarter search using parallel execution.Search query
Quarters to search in parallel
Number of chunks to retrieve per quarter. If None, uses config default
List[Dict[str, Any]] - Combined and ranked results from all quarters
_search_keywords(query, max_results=None, target_quarter=None)
Search using PostgreSQL full-text search with keywords.Search query to extract keywords from
Maximum results. If None, uses keyword_max_results from config (default: 10)
Optional quarter filter
List[Dict[str, Any]] - Keyword search results with ts_rank scores
_search_keywords_with_ticker(query, ticker, target_quarter=None)
Search using PostgreSQL full-text search with keywords and ticker filtering.Search query
Ticker symbol to filter by
Optional quarter filter
List[Dict[str, Any]] - Filtered keyword search results
Search Result Structure
Each search result is a dictionary with the following structure:Hybrid Search Configuration
The search engine combines vector and keyword search results:Enable/disable hybrid search. If False, returns empty results
Weight for vector search results (0-1)
Weight for keyword search results (0-1)
Minimum similarity threshold for results
Enable parallel execution of vector and keyword searches
SentenceTransformer model to use for embeddings
Usage Examples
- Basic Sync Search
- Async Multi-Query
- Multi-Quarter Search
Performance Characteristics
Search Speed
- Vector Search: Fast with proper database indexing (typically <100ms)
- Keyword Search: Very fast using PostgreSQL full-text search (<50ms)
- Hybrid Search: Combined time when parallel retrieval is enabled (~100ms)
Parallel Execution
Multi-Quarter
Quarters are searched in parallel using asyncio.gather
Vector + Keyword
When parallel_retrieval_enabled, both searches run simultaneously
Multi-Query
Multiple queries executed in parallel and deduplicated
Thread Pool
CPU-bound embedding operations run in thread pool to prevent blocking
Deduplication
Results are deduplicated by citation (chunk index) when merging:- Keeps the result with the best score (lowest distance)
- Prevents duplicate chunks in final results
- Applied in multi-query and follow-up searches
Logging
The search engine provides detailed logging:- Query embedding generation time
- Vector search results count
- Keyword search results count
- Result combination time
- Total search time
- Per-quarter search results
Error Handling
The search engine handles errors gracefully:- Embedding Errors: Logged and returns empty results
- Database Errors: Caught and logged, returns empty results
- Query Parsing Errors: Logged with warning, attempts fallback
- Multi-Quarter Failures: Individual quarter failures don’t stop other quarters
Related
RAGAgent
Main agent that orchestrates search operations
Agent
Top-level agent interface