Performance Optimization

Overview

Chronos-DFIR is engineered to handle massive forensic datasets (6GB+ files, 500K+ events) with responsive UI and minimal memory footprint. The platform achieves this through:

Streaming I/O for large file ingestion
Polars vectorized processing (zero pandas dependency)
Virtual DOM rendering with Tabulator.js
CSS GPU acceleration hints
Backend aggregation to minimize frontend computation

Target hardware: Apple Silicon M4 with ARM NEON SIMD and unified memory architecture. All optimizations leverage these capabilities.

Backend Performance

Streaming I/O for Large Files

All file operations for datasets exceeding 50MB use lazy loading with Polars scan_* methods. Files are never loaded entirely into memory until aggregated.

Streaming Upload (6GB+ Files)

Problem: Traditional await file.read() loads entire file into RAM before processing.Solution: Chunk-based streaming with SHA256 hash computation:

# app.py - Streaming upload handler
@app.post("/api/upload")
async def upload_file(file: UploadFile):
    save_path = os.path.join(UPLOAD_DIR, sanitize_filename(file.filename))
    hasher = hashlib.sha256()
    file_size = 0
    
    async with aiofiles.open(save_path, 'wb') as out_file:
        while content := await file.read(1024 * 1024):  # 1MB chunks
            await out_file.write(content)
            hasher.update(content)
            file_size += len(content)
    
    file_hash = hasher.hexdigest()
    return {
        "filename": file.filename,
        "size": file_size,
        "chain_of_custody": {"sha256": file_hash, "size": file_size}
    }

Benefits:

Zero extra I/O: Hash computed during upload, not as a separate read
Constant memory: 1MB buffer regardless of file size
Chain of custody: Forensic-grade hash available immediately

Reference: CLAUDE.md:19-20 (Streaming I/O rule)

Lazy File Parsing with Polars

Problem: Loading a 2GB CSV with pd.read_csv() or pl.read_csv() materializes the entire dataset.Solution: Use scan_csv() to return a LazyFrame:

# engine/ingestor.py
if ext == '.csv':
    lf = pl.scan_csv(
        file_path,
        infer_schema_length=0,  # Read all as strings
        ignore_errors=True,
        low_memory=True
    )
elif ext == '.parquet':
    lf = pl.scan_parquet(file_path)

Key Optimization: Queries are not executed until .collect() is called. Filters and aggregations are pushed down to the scan layer.

# GOOD: Filter before collecting (lazy execution)
result = lf.filter(pl.col('EventID') == 4624).collect(streaming=True)

# BAD: Collect first, then filter (materializes entire dataset)
df = lf.collect()
result = df.filter(pl.col('EventID') == 4624)

Reference: engine/ingestor.py:74-76 (Parquet/CSV lazy scan)

Streaming Aggregation

Use Case: Generate histogram buckets from 500K events without loading all rows.

# engine/analyzer.py - Streaming histogram aggregation
q_parsed = lf.lazy().select(['ts', 'Level', 'EventID'])

# Collect global stats with streaming mode
global_stats_df = q_parsed.select([
    pl.col('ts').min().alias('min_ts'),
    pl.col('ts').max().alias('max_ts'),
    pl.len().alias('count')
]).collect(streaming=True)  # <-- Streaming mode

# Bucket aggregation (also streaming)
histogram_df = q_parsed.group_by_dynamic(
    'ts',
    every=bucket_interval,
    label='left'
).agg([
    pl.len().alias('count'),
    pl.col('Level').mode().alias('dominant_level')
]).collect(streaming=True)

Benefits:

.collect(streaming=True) processes data in batches
Memory usage stays constant regardless of dataset size
Leverages Polars’ query optimizer for pushdown filters

Reference: engine/analyzer.py:91-95 (Streaming stats collection)

Hard Rule: All file I/O for datasets > 50MB must use streaming (scan_*/sink_*). Never use .read_csv() or .collect() on raw LazyFrames without filters.

Polars Vectorized Processing

Chronos-DFIR has zero pandas dependency in the core engine. All data transformations use Polars vectorized expressions.

No Pandas Policy

Why No Pandas?

Pandas is row-oriented → slow for large datasets
Polars is columnar (Apache Arrow) → 5-100x faster for DFIR operations
Pandas blocks the event loop → Polars supports lazy execution

v180 Pandas Elimination:

Before v180: 9 pandas imports in app.py (SQLite, Plist, whitespace CSV)
After v180: Zero pandas

Migrations:

# BEFORE (pandas)
df = pd.read_sql_query("SELECT * FROM events", conn)

# AFTER (Polars)
cursor.execute("SELECT * FROM events")
col_names = [desc[0] for desc in cursor.description]
rows = cursor.fetchall()
df = pl.DataFrame(
    {col_names[i]: [row[i] for row in rows] for i in range(len(col_names))},
    strict=False
)

Reference: CLAUDE.md:217-220 (Pandas elimination v180)

Vectorized Expressions (No Python Loops)

Anti-Pattern: Iterating over DataFrame rows

# BAD: Python loop (slow)
for row in df.iter_rows(named=True):
    if 'powershell' in row['Process'].lower():
        suspicious_procs.append(row['Process'])

# GOOD: Vectorized filter (fast)
suspicious_df = df.filter(
    pl.col('Process').cast(pl.Utf8).str.to_lowercase().str.contains('powershell')
)

Complex Example: Process top/rare analysis in engine/forensic.py

# engine/forensic.py:sub_analyze_identity_and_procs
# Filter meaningful processes (vectorized)
proc_counts = df.filter(
    pl.col(proc_col).is_not_null() &
    (pl.col(proc_col).cast(pl.Utf8).str.len_chars() > 0) &
    ~pl.col(proc_col).is_in(['-', 'N/A', 'null', 'None']) &
    ~pl.col(proc_col).cast(pl.Utf8).str.contains(r'^\d{1,5}$')  # Exclude pure numeric
).group_by(proc_col).agg(
    pl.len().alias('count')
).sort('count', descending=True)

top_procs = proc_counts.head(10).to_dicts()
rare_procs = proc_counts.filter(pl.col('count') == 1).head(10).to_dicts()

Performance: Polars processes 500K rows in ~50ms vs pandas ~2-3 seconds.Reference: CLAUDE.md:145-146 (v178 process filtering fix)

Async Threading for CPU-Bound Work

Problem: Polars operations are CPU-bound → block FastAPI’s event loopSolution: Wrap in asyncio.to_thread()

# app.py - Parallel forensic analysis
@app.post("/api/forensic_report/{filename}")
async def generate_forensic_report(filename: str):
    df = processed_files[filename]['df']
    
    # Run 9 CPU-bound tasks in parallel without blocking event loop
    task_results = await asyncio.gather(
        asyncio.to_thread(sub_analyze_timeline, df),
        asyncio.to_thread(sub_analyze_context, df),
        asyncio.to_thread(sub_analyze_hunting, df),
        asyncio.to_thread(sub_analyze_identity_and_procs, df),
        asyncio.to_thread(evaluate_sigma_rules, df, sigma_rules),
        asyncio.to_thread(scan_with_yara, df),
        asyncio.to_thread(build_correlation_chains, df),
        asyncio.to_thread(analyze_sessions, df),
        asyncio.to_thread(calculate_smart_risk_m4, df, sigma_hits)
    )
    return {"timeline": task_results[0], "context": task_results[1], ...}

Benefits:

FastAPI can handle other requests while Polars crunches data
Total analysis time: ~800ms for 38K events (vs 3-5s blocking)

Reference: CLAUDE.md:21 (Async-first rule)

Best Practice: Use .lazy() for all DataFrame operations, chain filters/aggregations, then .collect(streaming=True) once at the end.

Frontend Performance

Virtual DOM with Tabulator.js

Tabulator.js renders only visible rows (typically 50-100 at a time) even when the dataset has 500K+ events.

Remote Pagination Strategy

Configuration:

// static/js/grid.js
const table = new Tabulator('#data-table', {
    ajaxURL: `/api/data/${filename}`,
    pagination: true,
    paginationMode: 'remote',  // Server-side pagination
    paginationSize: 1000,      // 1000 rows per page
    ajaxParams: () => ({
        query: ChronosState.globalSearch,
        start_time: ChronosState.timeRange.start,
        end_time: ChronosState.timeRange.end,
        col_filters: JSON.stringify(ChronosState.columnFilters),
        selected_ids: JSON.stringify([...ChronosState.selectedIds])
    })
});

Backend Response:

# app.py - Remote pagination endpoint
@app.get("/api/data/{filename}")
async def get_data_page(filename: str, page: int = 1, size: int = 1000):
    df = processed_files[filename]['df']
    # Apply filters, then paginate
    filtered = _apply_standard_processing(df, query, col_filters, selected_ids)
    total = len(filtered)
    start = (page - 1) * size
    end = start + size
    page_data = filtered.slice(start, size).to_dicts()
    return {
        "last_page": math.ceil(total / size),
        "data": page_data,
        "total": total,
        "total_unfiltered": len(df)
    }

Performance: Only 1000 rows transferred per page → ~50KB JSON vs 50MB for full dataset.Reference: CLAUDE.md:89 (v177 remote sort fix)

Persistent Row Selection

Challenge: Checkbox selections must survive AJAX pagination.Solution: _persistentSelectedIds Set in grid.js

// static/js/grid.js
const _persistentSelectedIds = new Set();

table.on('rowSelected', (row) => {
    const id = row.getData()._id;
    _persistentSelectedIds.add(id);
    ChronosState.selectedIds.add(id);
});

table.on('rowDeselected', (row) => {
    if (_isReloading) return;  // Guard during table reload
    const id = row.getData()._id;
    _persistentSelectedIds.delete(id);
    ChronosState.selectedIds.delete(id);
});

Export Integration: When exporting, backend receives selected_ids array and filters accordingly.Reference: CLAUDE.md:286 (v180.7 persistent selection)

Batch Redraw for Column Operations

Problem: Hiding 50 empty columns with col.hide() triggers 50 full re-renders.Solution: Wrap in blockRedraw() / restoreRedraw()

// static/js/main.js - Hide empty columns
function toggleEmptyColumns() {
    const columns = table.getColumns();
    table.blockRedraw();  // Pause rendering
    
    columns.forEach(col => {
        const field = col.getField();
        if (isColumnEmpty(field)) {
            col.hide();
        } else {
            col.show();
        }
    });
    
    table.restoreRedraw();  // Single batch render
}

Performance: 50 columns = 1 render instead of 50 (20x faster).Reference: CLAUDE.md:86 (v177 toggleEmptyColumns fix)

CSS Performance Optimizations

Chronos-DFIR leverages modern CSS features to offload rendering to the GPU compositor.

content-visibility: auto

Purpose: Browser skips rendering offscreen elements until they’re scrolled into view.

/* static/chronos_v110.css */
.tabulator {
    content-visibility: auto;
    contain-intrinsic-size: 800px;  /* Placeholder height for scrollbar */
}

Benefits:

Initial page load: 60% faster (skips rendering 20+ offscreen cards)
Scroll performance: GPU handles visibility toggle

Reference: CLAUDE.md:148 (v178 CSS GPU hints)

will-change: transform

Purpose: Hints to browser that an element will be animated → promotes to GPU layer.

/* static/chronos_v110.css */
#chart-wrapper canvas {
    will-change: transform;
}

Correction (v179): Initially applied to #chart-wrapper (has display: none), moved to the actual canvas element.Caution: Only use on elements that actually animate. Overuse hurts performance.Reference: CLAUDE.md:179 (v179 will-change correction)

CSS Grid for Dashboard Cards

Before: Flexbox with manual spacingAfter: CSS Grid with gap

.dashboard-grid {
    display: grid;
    grid-template-columns: repeat(auto-fit, minmax(250px, 1fr));
    gap: 1rem;
    content-visibility: auto;  /* Skip offscreen cards */
}

Performance: Browser optimizes grid layout calculations in compositor thread.

CSS Performance Anti-Pattern: Avoid will-change on hidden elements (display: none). It has no effect and wastes GPU memory.

Asset Cache-Busting Mechanism

Chronos-DFIR uses automated cache-busting to prevent stale JavaScript/CSS after deployments.

Auto-Computed Version Hash

Implementation:

# app.py - Startup hash computation
import hashlib

def _compute_asset_hash():
    """MD5 hash of main JS + CSS files for automatic cache-busting."""
    files_to_hash = [
        os.path.join(STATIC_DIR, 'js', 'main.js'),
        os.path.join(STATIC_DIR, 'chronos_v110.css'),
    ]
    h = hashlib.md5()
    for fp in files_to_hash:
        if os.path.exists(fp):
            h.update(open(fp, 'rb').read())
    return h.hexdigest()[:8]

ASSET_VERSION = _compute_asset_hash()
logger.info(f"Asset version hash: {ASSET_VERSION}")

Template Usage (Jinja2):

<!-- templates/index.html -->
<link rel="stylesheet" href="/static/chronos_v110.css?v={{ v }}">
<script type="module" src="/static/js/main.js?v={{ v }}"></script>

Route Injection:

@app.get("/", response_class=HTMLResponse)
async def index(request: Request):
    return templates.TemplateResponse("index.html", {
        "request": request,
        "v": ASSET_VERSION  # Inject hash
    })

Reference: CLAUDE.md:133-143 (Auto-cachebust system)

Module Import Cache-Busting

Limitation: _compute_asset_hash() only hashes entry points (main.js, CSS). ES6 module imports use hardcoded version tags.

// static/js/main.js
import { renderTimeline } from './charts.js?v=185';
import { initGrid } from './grid.js?v=185';
import { setupFilters } from './filters.js?v=185';
import { exportData } from './actions.js?v=185';
import { ChronosState } from './state.js?v=185';

Manual Process: Increment ?v=XXX when any module changes.v181 Lesson: All v180.7 fixes were invisible to users because imports were cached at ?v=180. Server restart + manual bump to ?v=181 required.Reference: CLAUDE.md:330-332 (Cache bust lesson)

Best Practice: After modifying any JS module, increment the version tag in main.js imports before committing. Add a pre-commit hook to automate this.

Backend Aggregation Strategy

Chronos-DFIR pushes all heavy computation to the backend. The frontend receives pre-aggregated data ready for rendering.

Histogram Pre-Aggregation

Bad Pattern (client-side calculation):

// DON'T: Calculate stats in JavaScript
const peak = Math.max(...chartData.map(d => d.count));
const mean = chartData.reduce((sum, d) => sum + d.count, 0) / chartData.length;

Good Pattern (backend provides stats):

# app.py - Histogram endpoint
@app.get("/api/histogram/{filename}")
async def get_histogram(filename: str):
    histogram_data = await asyncio.to_thread(analyze_dataframe, df)
    return {
        "labels": histogram_data['labels'],
        "datasets": histogram_data['datasets'],
        "stats": {
            "peak": histogram_data['peak'],
            "mean": histogram_data['mean'],
            "trend": histogram_data['trend']  # alza/baja/estable
        }
    }

Frontend (v179 fix):

// static/js/charts.js - Use backend stats
function renderTimeline(data) {
    const peak = data.stats.peak;    // From backend
    const mean = data.stats.mean;    // From backend
    // No client-side reduce/Math.max needed
}

Reference: CLAUDE.md:180 (v179 backend stats preference)

Dashboard Card Aggregation

Endpoint: /api/forensic_report/{filename}Backend Computes:

Top 5 Event IDs (with labels)
Top 5 Tactics (MITRE ATT&CK)
Unique IPs, Users, Hosts counts
Sigma detection counts by severity (Critical, High, Medium, Low)
Risk score (0-100) with justification

Frontend: Receives JSON, renders cards. Zero calculation.

// static/js/main.js
async function loadDashboardCards() {
    const data = await fetch(`/api/forensic_report/${filename}`).then(r => r.json());
    document.getElementById('risk-score').textContent = data.risk_level;
    document.getElementById('unique-ips').textContent = data.context.unique_ips;
    // Just render, no computation
}

Reference: CLAUDE.md:287 (v180.7 dashboard refresh)

Performance Benchmarks

Real-World Test Case: 38K EVTX Events

Hardware: MacBook Pro M4 (32GB RAM, ARM NEON)

Operation	Time (v176)	Time (v185)	Improvement
File Upload (4.2GB)	18s	12s	33% faster
Initial Parse	2.8s	1.9s	32% faster
Sigma Evaluation (86 rules)	1.2s	0.8s	33% faster
Forensic Report (9 tasks)	3.5s	0.8s	77% faster
Histogram Render	420ms	180ms	57% faster
Export CSV (selected 500 rows)	1.1s	0.3s	73% faster
PDF Generation	8.5s	4.2s	51% faster

Key Optimizations:

v177: Streaming upload, fixed remote sort
v179: Async threading for forensic tasks
v180: Pandas elimination, app.py decomposition
v180.7: Persistent selection, batch redraw
v185: Chart debounce, CSV hex preservation

Profiling Tools

Chronos-DFIR includes built-in profiling for performance diagnostics.

Backend Profiling

Polars Query Plans:

# engine/analyzer.py
q = df.lazy().filter(...).group_by(...).agg(...)
print(q.explain())  # Show optimized query plan
result = q.collect(streaming=True)

Async Task Timing:

import time
start = time.perf_counter()
result = await asyncio.to_thread(sub_analyze_timeline, df)
logger.info(f"Timeline analysis: {time.perf_counter() - start:.3f}s")

Frontend Profiling

Chrome DevTools:

Performance tab → Record timeline during file load
Look for “Long Tasks” (> 50ms main thread blocks)
Check “Compositor” for GPU layer promotions

Tabulator Events:

table.on('renderStarted', () => console.time('render'));
table.on('renderComplete', () => console.timeEnd('render'));

System Architecture

Understand the data flow from ingestion to visualization

Multi-Agent Workflow

Learn about the 3-agent development protocol

Get Started

Core Features

Guides

Detection

Advanced

Performance Optimization

Overview

Backend Performance

Streaming I/O for Large Files

Polars Vectorized Processing

Frontend Performance

Virtual DOM with Tabulator.js

CSS Performance Optimizations

Asset Cache-Busting Mechanism

Backend Aggregation Strategy

Performance Benchmarks

Real-World Test Case: 38K EVTX Events

Profiling Tools

System Architecture

Multi-Agent Workflow

Build docs developers (and LLMs) love

Get Started

Core Features

Guides

Detection

Advanced

​Overview

​Backend Performance

​Streaming I/O for Large Files

​Polars Vectorized Processing

​Frontend Performance

​Virtual DOM with Tabulator.js

​CSS Performance Optimizations

​Asset Cache-Busting Mechanism

​Backend Aggregation Strategy

​Performance Benchmarks

​Real-World Test Case: 38K EVTX Events

​Profiling Tools

​Related Documentation

System Architecture

Multi-Agent Workflow

Build docs developers (and LLMs) love

Overview

Backend Performance

Streaming I/O for Large Files

Polars Vectorized Processing

Frontend Performance

Virtual DOM with Tabulator.js

CSS Performance Optimizations

Asset Cache-Busting Mechanism

Backend Aggregation Strategy

Performance Benchmarks

Real-World Test Case: 38K EVTX Events

Profiling Tools

Related Documentation