System Architecture

Metlo is built with a microservices architecture designed for scalability, reliability, and real-time API security monitoring. This page explains how the different components work together.

Architecture Diagram

┌─────────────────┐
│   Your API      │
│  Application    │
└────────┬────────┘
         │ Traffic
         ▼
┌─────────────────┐      ┌──────────────┐
│ Metlo Ingestor  │◄────►│   Redis      │
│   (port 8081)   │      │   Cache      │
└────────┬────────┘      └──────────────┘
         │
         ▼
┌─────────────────┐      ┌──────────────┐
│   PostgreSQL    │◄────►│   Metlo      │
│    Database     │      │   Backend    │
└─────────────────┘      │  (port 8080) │
         ▲               └──────┬───────┘
         │                      │
         │                      ▼
    ┌────┴────────┐      ┌──────────────┐
    │   Analyzer  │      │   Frontend   │
    │   Worker    │      │  (port 8000) │
    └─────────────┘      └──────────────┘
         ▲
         │
    ┌────┴────────┐
    │    Jobs     │
    │   Runner    │
    └─────────────┘

Core Components

1. Ingestor

Metlo Ingestor

Receives and processes API traffic from your applications in real time.

Port: 8081
Image: metlo/backend:latest
Command: yarn start-collector The ingestor is the entry point for all API traffic data. It:

Receives HTTP request/response data from Metlo agents in your applications
Validates and normalizes incoming traffic data
Performs initial processing and enrichment
Stores raw traffic data in PostgreSQL
Queues jobs for the analyzer via Redis

Key Features:

High-throughput traffic ingestion
Minimal latency impact on your APIs
Automatic batching and buffering
Handles traffic spikes gracefully

Configuration:

# docker-compose.yaml
ingestor:
  image: metlo/backend:latest
  command: yarn start-collector
  ports:
    - 8081:8081
  environment:
    - DB_URL=postgres://postgres:postgres@db:5432/metlo_api_security
    - REDIS_URL=redis://:eYVX7EwVmmxKPCDmwMtyKVge8oLd2t81@cache:6379
    - ENCRYPTION_KEY=${ENCRYPTION_KEY}
    - RUN_MIGRATION=true

The ingestor runs database migrations on startup when RUN_MIGRATION=true is set.

2. Backend API

Metlo Backend

Provides the REST API for the frontend dashboard and manages core business logic.

Port: 8080
Image: metlo/backend:latest
Technology: Node.js, Express, TypeORM The backend service handles:

User authentication and session management
API endpoint inventory management
Attack detection rule configuration
Security test execution
Alert and notification management
Data aggregation and analytics

Key Capabilities:

RESTful API for dashboard
User and role management (Enterprise)
Attack blocking rules
Security testing framework
Integration with CI/CD systems

Technology Stack:

{
  "runtime": "Node.js",
  "framework": "Express.js",
  "orm": "TypeORM",
  "session": "express-session with Redis",
  "validation": "OpenAPI validators",
  "testing": "@metlo/testing framework"
}

3. Frontend Dashboard

Metlo Frontend

Web-based UI for visualizing API security, managing endpoints, and configuring protection.

Port: 8000 (mapped from container port 3000)
Image: metlo/frontend:latest
Technology: Next.js, React, Chakra UI The frontend provides:

API Inventory Dashboard - Complete view of all discovered endpoints
Attack Monitoring - Real-time view of detected attacks and threats
Sensitive Data View - Identifies endpoints handling PII and sensitive data
Security Testing - Build and run security tests
Analytics - Traffic patterns, risk scores, and trends
Configuration - Manage detection rules and blocking policies

Technology Stack:

{
  "framework": "Next.js 13",
  "ui": "Chakra UI",
  "charts": "Chart.js",
  "workflow": "React Flow",
  "editor": "Monaco Editor",
  "data": "React Data Table Component"
}

4. Analyzer

Metlo Analyzer

Processes API traffic to detect attacks, identify patterns, and classify threats.

Image: metlo/jobrunner:latest
Command: yarn start-analyzer The analyzer is a worker pool that:

Analyzes API traffic for attack patterns
Detects SQL injection, XSS, BOLA, and other OWASP Top 10 vulnerabilities
Identifies sensitive data (PII, credentials, tokens)
Builds behavioral models of normal API usage
Generates attack signatures for blocking
Updates risk scores for endpoints

Detection Capabilities:

SQL Injection attempts
Cross-Site Scripting (XSS)
Broken Object Level Authorization (BOLA)
Broken Authentication
Security misconfiguration
Sensitive data exposure
XML External Entities (XXE)
Server-Side Request Forgery (SSRF)

Configuration:

analyzer:
  image: metlo/jobrunner:latest
  command: yarn start-analyzer
  environment:
    - DB_URL=postgres://postgres:postgres@db:5432/metlo_api_security
    - REDIS_URL=redis://:eYVX7EwVmmxKPCDmwMtyKVge8oLd2t81@cache:6379
    - NUM_WORKERS=${NUM_WORKERS}
    - ENCRYPTION_KEY=${ENCRYPTION_KEY}

Scale the analyzer by increasing NUM_WORKERS for high-traffic environments.

5. Jobs Runner

Metlo Jobs

Executes scheduled tasks and background jobs for maintenance and optimization.

Image: metlo/jobrunner:latest The jobs runner handles:

Scheduled Tasks - Periodic cleanup and maintenance
Endpoint Aggregation - Consolidates similar endpoints
Alert Generation - Creates notifications for detected issues
Data Retention - Manages data lifecycle policies
Report Generation - Compiles security reports
Integration Syncs - Updates external systems

Job Types:

Traffic aggregation and rollups
Alert rule evaluation
Data retention cleanup
Security test scheduling
Metrics calculation

6. Database (PostgreSQL)

PostgreSQL

Primary data store for API endpoints, traffic logs, attacks, and configuration.

Image: postgres:14.4-alpine
Port: 5432 (internal only) Stores:

API endpoint metadata and specifications
Raw and processed traffic logs
Detected attacks and alerts
User accounts and sessions
Security test definitions and results
Configuration and rules

Initialization:

-- init.sql
CREATE DATABASE metlo_api_security;

In production, use a managed PostgreSQL service (AWS RDS, Google Cloud SQL, Azure Database) for better reliability and automated backups.

7. Cache (Redis)

Redis Cache

High-performance cache for session data, job queues, and real-time state.

Image: redis:7.0.4
Port: 6379 (internal only) Used For:

Session storage (via connect-typeorm)
Job queue management (via bull)
Real-time attack state
Rate limiting data
Temporary aggregations
Cache for frequently accessed data

Data Flow

Traffic Ingestion Flow

Application Sends Traffic

Your application’s Metlo agent captures HTTP request/response data and sends it to the ingestor.

// Metlo agent in your app
app.use(MetloSDK({ host: "http://ingestor:8081" }));

Ingestor Processes

The ingestor validates, normalizes, and stores the traffic data in PostgreSQL, then queues analysis jobs in Redis.

Analyzer Detects Threats

The analyzer workers pick up jobs from Redis, analyze traffic for attack patterns, and update the database with findings.

Dashboard Displays

The frontend queries the backend API, which retrieves data from PostgreSQL to display in the dashboard.

Attack Detection Flow

1. Traffic arrives at Ingestor
   ↓
2. Stored in PostgreSQL
   ↓
3. Analysis job queued in Redis
   ↓
4. Analyzer picks up job
   ↓
5. Pattern matching against attack signatures
   ↓
6. If attack detected:
   - Store attack details in DB
   - Generate alert
   - Update endpoint risk score
   - Create blocking rule (if enabled)
   ↓
7. Frontend displays attack in dashboard

Deployment Architecture

Local Development

Use docker-compose.yaml for local development:

docker-compose up -d

All components run on a single machine with minimal resource requirements.

Production Deployment

For production, deploy components separately:

AWS Deployment

ECS/EKS for containers
RDS for PostgreSQL
ElastiCache for Redis
ALB for load balancing

GCP Deployment

GKE for containers
Cloud SQL for PostgreSQL
Memorystore for Redis
Cloud Load Balancing

Azure Deployment

AKS for containers
Azure Database for PostgreSQL
Azure Cache for Redis
Azure Load Balancer

Kubernetes

Use Helm charts
StatefulSets for stateful components
Services for networking
Ingress for external access

Scaling Considerations

Horizontal Scaling

Components that scale horizontally:

Ingestor: Add more instances behind a load balancer for high traffic
Backend: Scale API servers for more dashboard users
Analyzer: Increase worker count or add more analyzer instances
Jobs: Run multiple job runners for parallel task execution

Stateful components:

PostgreSQL: Use read replicas or sharding for large datasets
Redis: Use Redis Cluster for high availability

Resource Requirements

Minimum (Development):

2 CPU cores
4GB RAM
20GB disk

Recommended (Production):

Ingestor: 2 CPU, 4GB RAM per instance
Backend: 2 CPU, 4GB RAM per instance
Analyzer: 4 CPU, 8GB RAM per instance
Jobs: 2 CPU, 4GB RAM
PostgreSQL: 4 CPU, 16GB RAM, 100GB SSD
Redis: 2 CPU, 4GB RAM

Security

Encryption

All sensitive data is encrypted using the ENCRYPTION_KEY environment variable.

Metlo encrypts:

API keys and credentials
Sensitive data detected in traffic
User session data
Integration tokens

Authentication

The backend uses:

Session-based authentication with express-session
Sessions stored in Redis with encryption
CSRF protection
Role-based access control (Enterprise)

Network Security

Best practices:

Run components in a private network
Only expose frontend and ingestor ports
Use TLS/SSL for all external connections
Implement network policies in Kubernetes

Monitoring

Monitor these key metrics:

Ingestor: Requests per second, ingestion latency
Analyzer: Queue depth, processing time, detection rate
Database: Connection pool usage, query performance
Redis: Memory usage, eviction rate
System: CPU, memory, disk I/O

Next Steps

Deploy to AWS

Step-by-step guide for AWS deployment

Deploy to GCP

Step-by-step guide for GCP deployment

Configure Ingestors

Set up agents in your applications

Development Guide

Contribute to Metlo or customize for your needs

Getting Started

Deployment

Core Features

Testing

Configuration

​System Architecture

​Architecture Diagram

​Core Components

​1. Ingestor

Metlo Ingestor

​2. Backend API

Metlo Backend

​3. Frontend Dashboard

Metlo Frontend

​4. Analyzer

Metlo Analyzer

​5. Jobs Runner

Metlo Jobs

​6. Database (PostgreSQL)

PostgreSQL

​7. Cache (Redis)

Redis Cache

​Data Flow

​Traffic Ingestion Flow

​Attack Detection Flow

​Deployment Architecture

​Local Development

​Production Deployment

AWS Deployment

GCP Deployment

Azure Deployment

Kubernetes

​Scaling Considerations

​Horizontal Scaling

​Resource Requirements

​Security

​Encryption

​Authentication

​Network Security

​Monitoring

​Next Steps

Deploy to AWS

Deploy to GCP

Configure Ingestors

Development Guide

Build docs developers (and LLMs) love

System Architecture

Architecture Diagram

Core Components

1. Ingestor

2. Backend API

3. Frontend Dashboard

4. Analyzer

5. Jobs Runner

6. Database (PostgreSQL)

7. Cache (Redis)

Data Flow

Traffic Ingestion Flow

Attack Detection Flow

Deployment Architecture

Local Development

Production Deployment

Scaling Considerations

Horizontal Scaling

Resource Requirements

Security

Encryption

Authentication

Network Security

Monitoring

Next Steps