Arcana provides three core operations: ingest (add documents), search (retrieve relevant content), and ask (RAG question-answering).
Ingesting Documents Ingest adds documents to your vector store, automatically chunking and embedding the content.
Basic Text
With Metadata
With Source ID
From File
# Ingest text content { :ok , document} = Arcana . ingest ( "Your content here" , repo: MyApp . Repo )
PDF support requires a PDF parser. The default uses Poppler’s pdftotext. See the installation guide for setup instructions.
Using Collections Organize documents into collections for better organization and routing:
# Simple collection name { :ok , document} = Arcana . ingest ( "Product documentation" , repo: MyApp . Repo , collection: "products" ) # With description (helps Agent.select/2 route queries) { :ok , document} = Arcana . ingest ( "API reference" , repo: MyApp . Repo , collection: %{ name: "api" , description: "REST API endpoints and parameters" } )
Add descriptions to collections when using the Agent pipeline. The LLM uses these descriptions to route questions to the right collection.
Searching Arcana supports three search modes to match your use case.
Search Modes Semantic Search
Full-Text Search
Hybrid Search
Find content by meaning, not just keywords: { :ok , results} = Arcana . search ( "functional programming" , repo: MyApp . Repo )
Best for conceptual questions and natural language queries. Find exact keyword matches: { :ok , results} = Arcana . search ( "Elixir" , repo: MyApp . Repo , mode: :fulltext )
Best for finding specific terms, names, or code. Combine both approaches for better coverage: { :ok , results} = Arcana . search ( "Elixir patterns" , repo: MyApp . Repo , mode: :hybrid )
Recommended default for general-purpose search. Filtering Results # Limit and threshold { :ok , results} = Arcana . search ( "query" , repo: MyApp . Repo , limit: 5 , threshold: 0.7 # Minimum similarity score ) # Filter by source or collection { :ok , results} = Arcana . search ( "query" , repo: MyApp . Repo , source_id: "book-123" , collection: "products" )
Hybrid Search Weights (PgVector) When using the PgVector backend, you can customize how semantic and full-text scores are combined:
# Favor semantic similarity (default: 0.5 each) { :ok , results} = Arcana . search ( "Elixir patterns" , repo: MyApp . Repo , mode: :hybrid , semantic_weight: 0.7 , fulltext_weight: 0.3 ) # Favor keyword matches { :ok , results} = Arcana . search ( "specific_function_name" , repo: MyApp . Repo , mode: :hybrid , semantic_weight: 0.3 , fulltext_weight: 0.7 )
Weight configuration is only available with the PgVector backend. The memory backend uses Reciprocal Rank Fusion (RRF) for hybrid search.
Question Answering Use Arcana.ask/2 to combine search with an LLM for RAG-style question answering:
{ :ok , answer} = Arcana . ask ( "What is Elixir?" , repo: MyApp . Repo , llm: "openai:gpt-4o-mini" , limit: 5 )
Custom Prompts Control how the LLM generates answers:
custom_prompt = fn question, context -> context_text = Enum . map_join (context, " \n\n " , & &1 .text) """ You are a helpful assistant. Answer based only on the provided context. Be concise and cite specific passages. Context: #{ context_text } Question: #{ question } """ end { :ok , answer} = Arcana . ask ( "What is Elixir?" , repo: MyApp . Repo , llm: "openai:gpt-4o-mini" , prompt: custom_prompt )
Next Steps
LLM Integration Connect Arcana to OpenAI, Anthropic, and other LLM providers
Agentic RAG Build sophisticated multi-step RAG pipelines
Search Algorithms Understand how each search mode works under the hood
Dashboard Manage documents and test queries in the web UI