Indexing Pipeline

​

Pipeline Architecture

​

Phase 1: Scan Repository

• Recursively scans the repository starting from the root
• Filters out .git, node_modules, .gitnexus, and other ignored directories
• Collects file paths and sizes (used for chunking in Phase 3)
• No content is read yet — this is purely a directory walk

const scannedFiles = await walkRepositoryPaths(repoPath, (current, total, filePath) => {
  onProgress({
    phase: 'extracting',
    percent: Math.round((current / total) * 15),
    detail: filePath
  });
});

​

Phase 2: Structure

• Creates File nodes for all scanned files
• Creates Folder nodes for all directories
• Establishes CONTAINS relationships (Folder → File)
• No parsing or content analysis yet

processStructure(graph, allPaths);

​

Phase 3: Parsing

​

Chunked Processing

• Chunk size: 20MB of source code per chunk
• Files are read in chunks to avoid loading the entire codebase into memory
• Each chunk is parsed, processed, and then freed

const CHUNK_BYTE_BUDGET = 20 * 1024 * 1024; // 20MB

​

Tree-sitter Parsing

• 11 languages supported: TypeScript, JavaScript, Python, Java, C, C++, C#, Go, Rust, PHP, Swift
• Native bindings (CLI) or WASM (Web UI)
• Worker threads parallelize parsing across CPU cores

const chunkWorkerData = await processParsing(
  graph, chunkFiles, symbolTable, astCache,
  onProgress,
  workerPool // Optional: uses workers for parallel parsing
);

​

Symbol Extraction

• Functions (top-level function declarations)
• Classes (class definitions)
• Methods (functions inside classes)
• Interfaces (TypeScript/Java interfaces, Go interfaces)
• Enums, Variables, Structs, Traits, etc.

• Symbol nodes in the graph
• Symbol table for lookup during resolution
• AST cache for import/call resolution

​

Phase 4: Resolution

• import-processor.ts:processImports() — Resolves import statements
• call-processor.ts:processCalls() — Resolves function calls
• heritage-processor.ts:processHeritage() — Resolves inheritance (EXTENDS, IMPLEMENTS)

​

Import Resolution

​

Call Resolution

1. Find the caller: Use Tree-sitter to find the enclosing function for each call site
2. Find the callee: Look up the called function in the symbol table
3. Resolve across files: Use import statements to resolve calls to imported functions
4. Assign confidence:
   • Same file: 1.0
   • Import-resolved: 0.85
   • Fuzzy global match: 0.5 or 0.3

const findEnclosingFunction = (node, filePath, symbolTable) => {
  let current = node.parent;
  while (current) {
    if (FUNCTION_NODE_TYPES.has(current.type)) {
      // Found the caller function
      return symbolTable.lookupExact(filePath, funcName);
    }
    current = current.parent;
  }
  return null; // Top-level call
};

​

Heritage Resolution

• EXTENDS edges for class inheritance
• IMPLEMENTS edges for interface implementation
• Confidence: 1.0 (these are explicit declarations)

• IMPORTS, CALLS, EXTENDS, IMPLEMENTS edges in the graph
• All edges have confidence and reason properties

​

Phase 5: Communities

​

Leiden Algorithm

• Input: Graph of symbols connected by CALLS, EXTENDS, IMPLEMENTS edges
• Output: Community assignments for each symbol
• Modularity score: Measures how well-defined the communities are

const details = await leiden.detailed(graph, {
  resolution: isLarge ? 2.0 : 1.0,  // Higher resolution = more granular communities
  maxIterations: isLarge ? 3 : 0     // Limit iterations for large graphs
});

​

Heuristic Labels

const generateHeuristicLabel = (memberIds, nodePathMap) => {
  // Find most common parent folder
  const folderCounts = new Map();
  memberIds.forEach(nodeId => {
    const parts = nodePathMap.get(nodeId).split('/');
    const folder = parts[parts.length - 2];
    folderCounts.set(folder, (folderCounts.get(folder) || 0) + 1);
  });
  // Return most common folder, capitalized
};

​

Community Properties

• cohesion: Internal edge density (0-1) — what fraction of edges stay within the community
• symbolCount: Number of symbols in the community
• heuristicLabel: Auto-generated name

• Community nodes
• MEMBER_OF edges (symbol → community)

​

Phase 6: Processes

​

Entry Point Detection

const calculateEntryPointScore = (name, language, isExported, callerCount, calleeCount) => {
  let score = 0;
  // 1. Call ratio (calls many, called by few)
  const callRatio = calleeCount / Math.max(1, callerCount);
  if (callRatio > 2) score += 50;

  // 2. Export status (public API)
  if (isExported) score += 30;

  // 3. Name patterns (handle*, on*, *Controller, *Handler, main, etc.)
  if (/^(handle|on|process|execute)/i.test(name)) score += 40;

  return score;
};

​

Trace Algorithm

1. Start at entry point
2. Follow CALLS edges with confidence ≥ 0.5
3. Limit depth to 10 steps (configurable)
4. Limit branching to 4 paths per node (configurable)
5. Stop at terminal nodes (no outgoing calls)

const traceFromEntryPoint = (entryId, callsEdges, config) => {
  const queue = [[entryId, [entryId]]];
  const traces = [];

  while (queue.length > 0) {
    const [currentId, path] = queue.shift();
    const callees = callsEdges.get(currentId) || [];

    if (callees.length === 0) {
      traces.push([...path]); // Terminal node
    } else {
      callees.slice(0, config.maxBranching).forEach(calleeId => {
        if (!path.includes(calleeId)) { // Avoid cycles
          queue.push([calleeId, [...path, calleeId]]);
        }
      });
    }
  }
  return traces;
};

​

Deduplication

• Remove traces that are subsets of longer traces
• Keep only the longest path per entry→terminal pair

​

Process Properties

• processType: intra_community (single cluster) or cross_community (spans multiple clusters)
• stepCount: Number of steps in the trace
• communities: List of community IDs touched
• entryPointId / terminalId: First and last symbols in the chain

• Process nodes
• STEP_IN_PROCESS edges (symbol → process, with step number)

​

Search Indexing

• File FTS: Index file paths
• Function FTS: Index function names
• Class FTS: Index class names
• Method FTS: Index method names
• Interface FTS: Index interface names

​

Output

• Nodes: ~5,000 (files + symbols + communities + processes)
• Relationships: ~15,000 (calls + imports + memberships + steps)
• Communities: ~20-50 (functional areas)
• Processes: ~50-100 (execution flows)

​

Performance Characteristics

​

Next Steps