Observer Daemon

​

Overview

• Multi-tool support: Claude Code, Cursor, Codex CLI, Goose
• Incremental learning: observations accumulate during sessions
• Signal-based lifecycle: hooks notify daemon of events (compact, stop)
• Staleness detection: auto-finalizes idle sessions
• Singleton enforcement: only one daemon runs at a time

​

Architecture

IDE Sessions → Observer Daemon → CEMS Server → Memory Storage
     ↓              ↓                  ↓
  Hooks         Polling           LLM Analysis
               Adapters          (Gemini 2.5 Flash)

​

Components

{
  "session_id": "abc123...",
  "tool": "claude",
  "project_id": "org/repo",
  "source_ref": "project:org/repo",
  "last_observed_bytes": 50000,
  "last_observed_at": 1709251200.0,
  "observation_count": 3,
  "epoch": 0,
  "last_finalized_at": 0.0,
  "last_growth_seen_at": 1709251300.0,
  "is_done": false,
  "last_observed_message_id": 42
}

{
  "type": "compact",
  "ts": 1709251200.0,
  "tool": "claude"
}

• compact - Finalize current epoch, bump epoch number, continue watching
• stop - Finalize current epoch, mark session done

​

How It Works

​

1. Discovery

1. Polls each adapter for active sessions (max_age_hours=2)
2. Filters to sessions with recent activity
3. Loads or creates state for each session

• Claude/Cursor/Codex: Find JSONL files modified within 2 hours
• Goose: Query SQLite for sessions with recent messages

​

2. Processing

1. Check signals (compact/stop) → handle lifecycle
2. Check file growth → send incremental observation
3. Check staleness → auto-finalize idle sessions

​

3. Growth Detection

1. Phase 1: Cheap pre-filter
   • Check raw byte delta
   • Must exceed MIN_RAW_DELTA_BYTES = 10,000 bytes
   • Fast file stat, no I/O
2. Phase 2: Real extracted-text gate
   • Extract text content from new bytes
   • Must exceed MIN_EXTRACTED_CHARS = 3,000 chars (~750 tokens)
   • Ensures meaningful observations

• Phase 1 filters out tiny changes (typing, cursor movements)
• Phase 2 ensures enough content for LLM analysis
• Avoids wasteful API calls on incremental edits

​

4. Incremental Observations

1. Enrich session metadata (project, git branch, cwd)
2. Extract new text content since last observation
3. Build project context string
4. Send to server via /api/session/summarize:
   {
     "content": "extracted text",
     "session_id": "abc123",
     "mode": "incremental",
     "epoch": 0,
     "session_tag": "session:abc123",
     "project_context": "org/repo (main) — /path/to/project",
     "source_ref": "project:org/repo"
   }
   
5. Server uses Gemini 2.5 Flash to extract observations
6. Update state: bytes, timestamp, observation_count

• “User deploys via Coolify”
• “Project uses PostgreSQL with pgvector”
• “Testing: RSpec for Ruby, pytest for Python”
• “Architecture: microservices with message queue”
• “Workflow: PR review required before merge”

​

5. Signal Handling

1. Finalize current epoch (if observations exist)
2. Bump epoch number: state.epoch += 1
3. Continue watching for new activity

• Epoch 0: Initial session observations
• Compact → Finalize epoch 0 doc
• Epoch 1: Post-compact observations
• Compact → Finalize epoch 1 doc
• Etc.

1. Finalize current epoch (if observations exist)
2. Mark session done: state.is_done = True
3. Stop watching this session

1. Extract any remaining new content
2. Send to server with mode: "finalize"
3. Server generates comprehensive final summary
4. Update last_finalized_at timestamp
5. Clear signal file

​

6. Staleness Detection

1. Check: time.time() - state.last_growth_seen_at > 300
2. Trigger auto-finalization
3. Mark session done

• Handles sessions without hooks (e.g., crashed IDE)
• Ensures observations are eventually finalized
• Prevents zombie sessions from clogging the queue

​

Installation & Setup

curl -fsSL https://getcems.com/install.sh | bash

uv tool install cems

• cems - Main CLI
• cems-server - Server component
• cems-observer - Observer daemon

​

Running the Daemon

​

Manual Start

cems-observer

python -m cems.observer

• --once - Run one cycle and exit (for testing)
• --verbose, -v - Enable debug logging

​

Automatic Start

1. Check if observer is already running (via PID file)
2. If not, spawn daemon in background
3. Write PID to ~/.cems/observer/daemon.pid
4. Continue hook execution

​

Singleton Enforcement

1. Daemon acquires exclusive file lock on ~/.cems/observer/daemon.lock
2. If lock fails, another daemon is already running → exit
3. Lock is held for the lifetime of the process
4. On exit, lock is released automatically

pgrep -f "cems[.-]observer"  # Find all observer processes
kill -TERM <pid>             # Kill stale ones

​

Configuration

​

Credentials

1. ~/.cems/credentials (preferred)
2. Environment variables (fallback)

# ~/.cems/credentials
CEMS_API_URL=https://cems.example.com
CEMS_API_KEY=cems_usr_...

export CEMS_API_URL=https://cems.example.com
export CEMS_API_KEY=cems_usr_...

​

Thresholds

MIN_RAW_DELTA_BYTES = 10_000    # Phase 1: raw byte pre-filter
MIN_EXTRACTED_CHARS = 3_000     # Phase 2: extracted text gate (~750 tokens)
POLL_INTERVAL = 30              # Polling interval (seconds)
STALE_THRESHOLD = 300           # Staleness threshold (5 minutes)

• Lower MIN_EXTRACTED_CHARS → more frequent observations (higher API costs)
• Higher MIN_EXTRACTED_CHARS → fewer observations (may miss details)
• Lower POLL_INTERVAL → more responsive (higher CPU usage)
• Higher POLL_INTERVAL → less responsive (lower CPU usage)

​

Storage

~/.cems/observer/
├── daemon.pid              # Current daemon PID
├── daemon.lock             # Singleton lock file
├── daemon.log              # Log file (if file logging enabled)
├── signals/                # Signal files from hooks
│   ├── abc123.json
│   └── def456.json
└── *.json                  # Session state files
    ├── abc123.json
    └── def456.json

• Old: ~/.claude/observer/
• New: ~/.cems/observer/

​

Monitoring

​

Check Daemon Status

# Check if daemon is running
ps aux | grep cems-observer

# Check PID file
cat ~/.cems/observer/daemon.pid

# Check logs (if file logging enabled)
tail -f ~/.cems/observer/daemon.log

​

Test One Cycle

cems-observer --once

Observations triggered: 3

​

Verbose Mode

cems-observer --verbose

Observer daemon started (PID 12345, polling every 30s)
Adapters: claude, cursor, codex, goose
CEMS API: https://cems.example.com
Thresholds: raw=10000B, extracted=3000chars
Staleness threshold: 300s

claude: 2 sessions (abc123ab, def456de)
Cycle 0: 2 observation(s) triggered

goose: 1 sessions (ghi789gh)
Cycle 1: 1 observation(s) triggered

​

Stop Daemon

kill -TERM $(cat ~/.cems/observer/daemon.pid)

kill -9 $(cat ~/.cems/observer/daemon.pid)

​

State Management

​

State File Format

{
  "session_id": "abc123def456...",
  "tool": "claude",
  "project_id": "org/repo",
  "source_ref": "project:org/repo",
  "last_observed_bytes": 50000,
  "last_observed_at": 1709251200.0,
  "observation_count": 3,
  "session_started": 1709250000.0,
  "epoch": 0,
  "last_finalized_at": 0.0,
  "last_growth_seen_at": 1709251300.0,
  "is_done": false,
  "last_observed_message_id": 42
}

• session_id - UUID from IDE
• tool - claude, cursor, codex, or goose
• project_id - org/repo format
• source_ref - “project:org/repo” for memory tagging
• last_observed_bytes - File offset for next extraction
• last_observed_at - Timestamp of last observation
• observation_count - Number of observations sent
• session_started - Session start timestamp
• epoch - Epoch number (bumped on compact)
• last_finalized_at - Timestamp of last finalization
• last_growth_seen_at - Staleness tracking
• is_done - Session complete flag
• last_observed_message_id - SQLite adapter watermark

​

State Cleanup

find ~/.cems/observer -name "*.json" -mtime +7 -delete

​

Epoch Model

• Initial session observations
• Document ID: session:abc123

• Post-compact observations
• Document ID: session:abc123:e1

• Next epoch observations
• Document ID: session:abc123:e2

• Allows finalization mid-session (during compacts)
• Keeps observation documents focused and coherent
• Prevents unbounded document growth
• Each epoch gets its own summary and memory tags

def session_tag(session_id: str, epoch: int = 0) -> str:
    tag = f"session:{session_id[:12]}"
    if epoch > 0:
        tag += f":e{epoch}"
    return tag

• session:abc123def456
• session:abc123def456:e1
• session:abc123def456:e2

​

Adapter Details

​

Claude Code Adapter

1. Scan ~/.claude/projects/
2. Find directories with transcript.jsonl
3. Filter by modification time (< 2 hours)

1. Read JSONL from last observed byte offset
2. Parse each line as JSON
3. Extract user messages and assistant responses
4. Format as conversation

• project_id from git remote in project directory
• git_branch from git status
• cwd from project directory path

​

Cursor Adapter

1. Scan ~/.cursor/
2. Find directories with transcript.jsonl
3. Filter by modification time

• Same as Claude Code adapter

​

Codex Adapter

1. Scan ~/.codex/transcripts/
2. Find *.jsonl files
3. Filter by modification time

• Same as Claude Code adapter

​

Goose Adapter

1. Scan ~/.config/goose/sessions/
2. Find *.db SQLite files
3. Query for sessions with recent messages

1. Query: SELECT * FROM messages WHERE id > ? ORDER BY id
2. Use last_observed_message_id as watermark
3. Extract message content
4. Format as conversation

session.extra.setdefault("last_observed_message_id", state.last_observed_message_id)

​

Error Handling

​

Consecutive Failures

1. Log warning: “N consecutive failures, backing off”
2. Switch to backoff interval: 300s (5 minutes)
3. Continue polling at reduced rate
4. Reset counter on successful cycle

• Server unavailable
• Network issues
• API rate limiting
• Authentication failures

​

Auth Failures

RuntimeError: Auth failed (401): check CEMS_API_KEY

1. Verify credentials in ~/.cems/credentials
2. Test with cems health
3. Restart daemon

​

Server Unavailable

• Connection timeout errors
• “Connection refused” messages

1. Check server status
2. Verify CEMS_API_URL is correct
3. Check network connectivity
4. Daemon will retry automatically

​

Corrupted State

• JSON decode errors in logs
• Observations re-sent from byte 0

rm ~/.cems/observer/<session_id>.json

​

Performance

​

Resource Usage

• State reads: ~10-20 per cycle
• State writes: ~2-5 per cycle
• Transcript reads: varies by session activity

• Observation API calls: 1-5 per cycle (when sessions are active)
• Payload size: 3KB - 50KB per observation

​

Optimization

• File stat: ~1ms
• File read + extraction: ~10-100ms

​

Troubleshooting

​

Observer Not Running

ps aux | grep cems-observer
cat ~/.cems/observer/daemon.pid

cems-observer

​

No Observations Being Created

cems-observer --once --verbose

• File growth below threshold (< 10KB raw, < 3K chars extracted)
• Session already marked done
• Signal file stuck (old compact/stop signal)

rm ~/.cems/observer/signals/*.json  # Clear stale signals

​

Observations Re-sent from Start

rm ~/.cems/observer/<session_id>.json

​

Duplicate Daemons

pkill -f cems-observer
cems-observer

​

Related Pages

• Skills & Commands - Skills and slash commands reference
• CLI Usage - Command-line interface reference
• Memory Management - Memory lifecycle and best practices