Skip to main content

​
Function Signature

search(ctx, opts \\ [])
Executes search and populates results in the context.

​
Purpose

Uses sub_questions if present (from decompose step), otherwise uses the original question. Searches the selected collections and populates results.

​
Parameters

​
ctx
Arcana.Agent.Context
required
The agent context from the pipeline
​
opts
Keyword.t()
Options for the search step

​
Options

​
searcher
module | function
Custom searcher module or function (default: Arcana.Agent.Searcher.Arcana)
  • Module: Must implement search/3 callback
  • Function: Signature fn question, collection, opts -> {:ok, chunks} | {:error, reason} end
​
collection
String.t()
Single collection name to searchTakes precedence over :collections and ctx.collections.
​
collections
list(String.t())
List of collection names to searchTakes precedence over ctx.collections.
​
self_correct
boolean
Enable self-correcting search (default: false)Note: This option is parsed but not currently implemented in the search step. Use reason/2 for multi-hop reasoning instead.
​
max_iterations
integer
Max retry attempts for self-correct (default: 3)Note: Currently unused. See reason/2 for iterative search.
​
sufficient_prompt
function
Custom prompt function fn question, chunks -> prompt_string endNote: Currently unused. See reason/2 for sufficiency evaluation.
​
rewrite_prompt
function
Custom prompt function for query rewritingNote: Currently unused. See reason/2 for iterative search.

​
Context Updates

​
results
list(map)
Search results. Each result contains:
  • question - The query that was searched
  • collection - The collection that was searched
  • chunks - List of retrieved chunks with id, text, and score

​
Collection Selection Priority

Collections are determined in this order:
  1. :collection option (single collection)
  2. :collections option (multiple collections)
  3. ctx.collections (set by select/2)
  4. Fallback: ["default"]

​
Examples

ctx
|> Arcana.Agent.search()  # Uses "default" collection
|> Arcana.Agent.answer()

ctx.results
# => [
#   %{
#     question: "What is Elixir?",
#     collection: "default",
#     chunks: [
#       %{id: 1, text: "Elixir is a functional...", score: 0.89},
#       %{id: 5, text: "Elixir runs on the BEAM...", score: 0.85}
#     ]
#   }
# ]

​
Search Specific Collection

ctx
|> Arcana.Agent.search(collection: "technical_docs")
|> Arcana.Agent.answer()

​
Search Multiple Collections

ctx
|> Arcana.Agent.search(collections: ["docs", "faq"])
|> Arcana.Agent.answer()

ctx.results
# => [
#   %{question: "What is Elixir?", collection: "docs", chunks: [...]},
#   %{question: "What is Elixir?", collection: "faq", chunks: [...]}
# ]

​
With LLM-Based Collection Selection

ctx
|> Arcana.Agent.select(collections: ["docs", "api", "guides"])
|> Arcana.Agent.search()  # Uses collections from select step
|> Arcana.Agent.answer()

​
With Decomposition

ctx = Arcana.Agent.new("How do GenServer and Agent differ?")
|> Arcana.Agent.decompose()
|> Arcana.Agent.search()

ctx.sub_questions
# => ["What is GenServer?", "What is Agent?", "Differences"]

ctx.results
# => [
#   %{question: "What is GenServer?", collection: "default", chunks: [...]},
#   %{question: "What is Agent?", collection: "default", chunks: [...]},
#   %{question: "Differences", collection: "default", chunks: [...]}
# ]

​
With Query Expansion

ctx
|> Arcana.Agent.expand()   # Adds related terms
|> Arcana.Agent.search()   # Searches with expanded query

​
Complete Pipeline

ctx
|> Arcana.Agent.rewrite()                       # Clean input
|> Arcana.Agent.select(collections: ["docs", "api"])
|> Arcana.Agent.expand()                        # Add synonyms
|> Arcana.Agent.decompose()                     # Split into sub-questions
|> Arcana.Agent.search()                        # Search all combinations
|> Arcana.Agent.answer()

# If decompose creates 2 sub-questions and select chooses 2 collections:
# Total searches = 2 sub-questions × 2 collections = 4 searches

​
Custom Searcher Module

defmodule MyApp.ElasticsearchSearcher do
  @behaviour Arcana.Agent.Searcher

  @impl true
  def search(question, collection, opts) do
    case Elasticsearch.search(collection, question, opts) do
      {:ok, results} ->
        chunks =
          Enum.map(results, fn hit ->
            %{id: hit.id, text: hit.source.content, score: hit.score}
          end)

        {:ok, chunks}

      {:error, reason} ->
        {:error, reason}
    end
  end
end

# Usage
ctx
|> Arcana.Agent.search(searcher: MyApp.ElasticsearchSearcher)

​
Inline Searcher Function

ctx
|> Arcana.Agent.search(
  searcher: fn question, collection, opts ->
    # Custom search logic
    {:ok, my_search_function(question, collection, opts)}
  end
)

defmodule MyApp.HybridSearcher do
  @behaviour Arcana.Agent.Searcher

  @impl true
  def search(question, collection, opts) do
    # Combine vector and keyword search
    with {:ok, vector_results} <- vector_search(question, collection, opts),
         {:ok, keyword_results} <- keyword_search(question, collection, opts) do
      merged = merge_and_deduplicate(vector_results, keyword_results)
      {:ok, merged}
    end
  end
end

​
Skip Retrieval

If gate/2 sets skip_retrieval: true, search returns empty results: 
ctx
|> Arcana.Agent.gate()     # Sets skip_retrieval: true for "2+2"
|> Arcana.Agent.search()   # Returns empty results
|> Arcana.Agent.answer()   # Answers without context

ctx.results
# => []

​
Custom Searcher Behaviour

defmodule Arcana.Agent.Searcher do
  @callback search(
              question :: String.t(),
              collection :: String.t(),
              opts :: Keyword.t()
            ) ::
              {:ok, [chunk :: map()]} | {:error, term()}
end
Chunks should have at minimum:
  • id - Unique identifier
  • text - The chunk content
  • score - Relevance score (optional)

​
Search Options Passed to Searcher

The searcher receives these options from context: 
[
  repo: ctx.repo,
  limit: ctx.limit,      # Default: 5
  threshold: ctx.threshold  # Default: 0.5
]

​
Telemetry Event

Emits [:arcana, :agent, :search] with metadata: 
# Start metadata
%{
  question: ctx.question,
  sub_questions: ctx.sub_questions,
  collections: ["docs", "api"],
  searcher: Arcana.Agent.Searcher.Arcana
}

# Stop metadata
%{
  result_count: 4,        # Number of search operations
  total_chunks: 20        # Total chunks retrieved
}

​
Performance Considerations

  • Total searches = sub_questions × collections
  • With 3 sub-questions and 2 collections = 6 searches
  • Consider limiting sub-questions and collections
  • Use reason/2 for iterative search instead of upfront decomposition

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us