Skip to main content

​
Architecture Overview

The JARVIS backend is built with FastAPI and follows a modular pipeline architecture for processing captured media, identifying faces, enriching person data, and synthesizing intelligence reports.

​
Core Components

​
Technology Stack

FastAPI

Async Python web framework for high-performance APIs

MediaPipe

Google’s real-time face detection (BlazeFace model)

Browser Use

Cloud browser agents for deep research across social platforms

Claude/Gemini

LLM synthesis for intelligence report generation

​
Application Initialization

The backend initializes all services during startup with graceful degradation:
backend/main.py
# Build pipeline components
detector = MediaPipeFaceDetector()
embedder = ArcFaceEmbedder()

# Database: use Convex when configured, else in-memory
convex_gw = ConvexGateway(settings)
db_gateway = convex_gw if convex_gw.configured else InMemoryDatabaseGateway()

# Face search: PimEyes (primary) + reverse image search (fallback)
face_searcher = FaceSearchManager(settings)

# Enrichment + research + synthesis (None when API keys missing)
exa_client = ExaEnrichmentClient(settings) if settings.exa_api_key else None
orchestrator = ResearchOrchestrator(settings) if settings.browser_use_api_key else None
synthesis_engine = AnthropicSynthesisEngine(settings) if settings.anthropic_api_key else None
synthesis_fallback = GeminiSynthesisEngine(settings) if settings.gemini_api_key else None

​
Lifespan Management

The application uses FastAPI’s lifespan context manager for startup/shutdown:
backend/main.py
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncIterator[None]:
    logger.info(
        "JARVIS started — det={} emb={} db={} face_search={} exa={} deep_researcher={} synth={} (primary) synth_fallback={} supermemory={}",
        detector.__class__.__name__,
        embedder.__class__.__name__,
        db_gateway.__class__.__name__,
        face_searcher is not None,
        exa_client is not None,
        deep_researcher is not None,
        synthesis_engine.__class__.__name__ if synthesis_engine else None,
        synthesis_fallback.__class__.__name__ if synthesis_fallback else None,
        supermemory_client is not None,
    )
    if telegram_bot:
        await telegram_bot.start()
    yield
    if telegram_bot:
        await telegram_bot.stop()
    if supermemory_client:
        await supermemory_client.close()
    logger.info("JARVIS shutting down")

​
Core API Endpoints

​
Health Check

backend/main.py
@app.get("/api/health", response_model=HealthResponse)
async def health() -> HealthResponse:
    return HealthResponse(
        status="ok",
        environment=settings.environment,
        services=settings.service_flags(),
    )

​
Service Status

Returns configuration status for all integrated services:
backend/main.py
@app.get("/api/services", response_model=list[ServiceStatus])
async def services() -> list[ServiceStatus]:
    descriptions = {
        "convex": "Real-time board subscriptions and mutations",
        "mongodb": "Persistent raw captures and dossiers",
        "exa": "Fast pass research and person lookup",
        "browser_use": "Deep research browser agents",
        "openai": "Transcription and fallback LLM integrations",
        "anthropic": "Primary synthesis model (Claude)",
        "gemini": "Fallback vision and synthesis model when Anthropic unavailable",
        "laminar": "Tracing and evaluation telemetry",
        "telegram": "Glasses-side media intake",
        "pimeyes_pool": "Rotating account pool for identification",
    }
    flags = settings.service_flags()
    return [
        ServiceStatus(name=name, configured=configured, notes=descriptions.get(name))
        for name, configured in flags.items()
    ]

​
CORS Configuration

The application configures CORS to allow the frontend to make cross-origin requests:
backend/main.py
app.add_middleware(
    CORSMiddleware,
    allow_origins=[settings.frontend_origin],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

​
Logging Configuration

All backend logs are written to both stdout and a file for debugging:
backend/main.py
logger.add("/tmp/jarvis_backend.log", rotation="10 MB", level="DEBUG",
           format="{time:HH:mm:ss.SSS} | {level:<7} | {name}:{function}:{line} | {message}")
Logs can be tailed with: tail -f /tmp/jarvis_backend.log

​
Service Flags

The backend exposes which services are configured via environment variables: 
# Check if services are available
flags = settings.service_flags()
if flags["browser_use"]:
    # Deep research available
    orchestrator = ResearchOrchestrator(settings)

​
Next Steps

Capture Service

Learn how media uploads are processed

Identification Pipeline

Explore face detection and search

Agent Orchestration

Understand the research agent system

Synthesis Engine

See how LLMs generate intelligence reports

Build docs developers (and LLMs) love

Get started for freeTalk to us