Skip to main content
Metadata filtering applies structured constraints to vector search results, enabling you to narrow results by fields like category, date, author, or custom attributes while still ranking by semantic relevance.

​
Overview

Vector search finds semantically similar content but cannot filter by structured properties. Metadata filtering adds boolean predicates (equals, range, contains) to retrieve precisely targeted documents.
Metadata filtering is essential for production RAG applications when you need queries like “find documents about ML published in 2024” or “retrieve technical articles by author X”.

​
How it works

The metadata filtering pipeline follows these steps:
  1. Query embedding - Convert query text to vector representation
  2. Filtered vector search - Execute similarity search with metadata constraints
  3. Post-processing - Apply additional client-side filters if configured
  4. RAG generation - Generate answer using filtered documents (optional)

​
Supported operators

VectorDB supports the following filter operators across all databases:
OperatorDescriptionExample
equalsExact matchcategory = "electronics"
not_equalsNot equalstatus != "archived"
gtGreater thanprice > 100
gteGreater than or equaldate >= "2024-01-01"
ltLess thanscore < 0.5
lteLess than or equalrating <= 4.5
inValue in listcategory in ["tech", "science"]
not_inValue not in listauthor not in ["user1", "user2"]
containsSubstring match (case-insensitive)title contains "machine"
startswithPrefix match (case-insensitive)name startswith "Dr"
endswithSuffix match (case-insensitive)filename endswith ".pdf"
String operators (contains, startswith, endswith) are case-insensitive for consistent behavior across databases.

​
Database-specific syntax

Each database uses its own native filter format: 
# Pinecone uses JSON filter syntax
filters = {
    "$and": [
        {"category": {"$eq": "technical"}},
        {"date": {"$gte": "2024-01-01"}}
    ]
}

​
Configuration

Define metadata filters in your pipeline configuration: 
filters:
  conditions:
    - field: "category"
      value: "technical"
      operator: "equals"
    - field: "price"
      value: 500
      operator: "lt"

​
Usage example

from vectordb.langchain.metadata_filtering.search.pinecone import (
    PineconeMetadataFilteringSearchPipeline,
)

pipeline = PineconeMetadataFilteringSearchPipeline("config.yaml")

# Search with filters
results = pipeline.search(
    "machine learning frameworks",
    top_k=10,
    filters={"category": {"$eq": "technical"}},
)

print(f"Found {len(results['documents'])} documents")
for doc in results["documents"]:
    print(f"- {doc.metadata['title']} (category: {doc.metadata['category']})")

​
Performance optimization

​
Selectivity analysis

Filter order matters for query performance. VectorDB includes selectivity analysis to optimize filter execution: 
# High selectivity filters (fewer matches) should run first
filters = [
    {"user_id": "specific-user"},  # High selectivity
    {"category": "news"},          # Lower selectivity
]

​
Pre-filter vs post-filter

Databases apply filters at different stages:
  • Pre-filter - Filter before vector search (faster, smaller search space)
  • Post-filter - Filter after vector search (preserves ranking quality)
Use pre-filtering for highly selective filters (user_id, tenant_id) and post-filtering for broader criteria (category, date ranges).

​
Timing metrics

Track filter performance with built-in timing metrics: 
results = pipeline.search(query, filters=filters)

print(f"Filter time: {results['metrics']['filter_time_ms']}ms")
print(f"Search time: {results['metrics']['search_time_ms']}ms")
print(f"Total time: {results['metrics']['total_time_ms']}ms")

JSON indexing

Filter by nested JSON paths

Namespaces

Logical data partitioning

Multi-tenancy

Tenant-isolated retrieval

Semantic search

Vector similarity search

Build docs developers (and LLMs) love

Get started for freeTalk to us