Infrastructure Overview

Architecture

Fluxer’s infrastructure is built on a modern, containerized stack designed for scalability, reliability, and operational excellence.

Core Infrastructure Components

Database

Cassandra 5.0 - Primary data store for messages, users, guilds, and relationships with automatic snapshots and encrypted backups

Caching & State

Valkey 8.0 - Redis-compatible in-memory store for sessions, rate limiting, and real-time state management

Message Queue

NATS JetStream - High-performance messaging for event distribution and service communication

Monitoring

SigNoz - OpenTelemetry-native observability platform for traces, metrics, and logs

Infrastructure Services

Data Layer

Service	Technology	Purpose
cassandra	Cassandra 5.0	Primary database for persistent data
cassandra-backup	Custom backup service	Automated encrypted backups to B2 object storage
valkey	Valkey 8.0 Alpine	In-memory cache and session store
nats_jetstream	NATS 2 Alpine	Message queue with persistence
nats_core	NATS 2 Alpine	Core messaging for real-time events

Observability Stack

Service	Technology	Purpose
signoz	SigNoz v0.108.0	APM and observability platform
otel-collector	SigNoz OTel Collector	OpenTelemetry data collection
clickhouse	ClickHouse 25.5.6	Time-series metrics storage
zookeeper	Zookeeper 3.7.1	Coordination for ClickHouse

Media & Proxies

Service	Technology	Purpose
LiveKit	LiveKit v1.9.11	WebRTC SFU for voice/video
Caddy	Caddy v2.10.2	Reverse proxy with auto TLS
coturn	coturn	TURN/STUN for NAT traversal
nginx	nginx	Static asset serving

Supporting Services

ClamAV - Malware Scanning

Antivirus engine for scanning uploaded files and attachments in real-time.

On-access scanning for media uploads
Daily signature updates
Quarantine and notification system

Weblate - Localization

Collaborative translation platform for community-driven internationalization.

Web-based translation interface
Git integration for version control
Translation memory and terminology

Ghost Blog - Marketing

Headless CMS for official blog and documentation.

SEO-optimized content delivery
Custom themes and layouts
Newsletter integration

Turborepo Cache - Build Optimization

Remote caching server for Turborepo monorepo builds.

Shared build cache across CI/CD
Significant build time reduction
S3-compatible storage backend

Deployment Model

Docker Swarm

Fluxer uses Docker Swarm for container orchestration, providing:

Service replication - Horizontal scaling with replica sets
Rolling updates - Zero-downtime deployments
Health checks - Automatic container restart on failure
Overlay networking - Secure inter-service communication
Secret management - Encrypted configuration and credentials

# Initialize swarm (if not already)
docker swarm init

# Create shared network
docker network create -d overlay fluxer-shared

# Deploy a service stack
cd fluxer_devops/cassandra
docker stack deploy -c compose.yaml fluxer-cassandra

Resource Allocation

Production Specifications

Cassandra
Valkey
OTel Collector

resources:
  limits:
    cpus: '6'
    memory: 64G
  reservations:
    cpus: '4'
    memory: 48G

Heap Size: 32GB (MAX_HEAP_SIZE)
Storage: Local SSD volumes for optimal I/O
Ulimits: 100k nofile for connection handling

resources:
  limits:
    cpus: '2'
    memory: 4G
  reservations:
    cpus: '1'
    memory: 2G

Persistence: AOF + RDB snapshots
Replication: Master-replica setup supported

deploy:
  replicas: 3

Batch Size: 10k events per batch
Flush Interval: 10s timeout
Ports: 4317 (gRPC), 4318 (HTTP)

Network Configuration

Port Allocations

Port(s)	Service	Protocol	Purpose
9042	Cassandra	TCP	CQL native protocol
6379	Valkey	TCP	Redis protocol
4222-4223	NATS	TCP	NATS client connections
8222	NATS	HTTP	NATS monitoring
7880-7881	LiveKit	TCP	WebRTC signaling
50000-60000	LiveKit	UDP	WebRTC media
3478	coturn	UDP	TURN/STUN
4317-4318	OTel	TCP	OpenTelemetry ingestion
8080	SigNoz	HTTP	Observability UI

Overlay Network

All services communicate over the fluxer-shared overlay network:

docker network create \
  --driver overlay \
  --subnet 10.20.0.0/16 \
  --attachable \
  fluxer-shared

Configuration Management

Configuration is managed through:

Environment files - /etc/fluxer/*.env for service-specific config
Docker configs - Immutable configuration objects in Swarm
Docker secrets - Encrypted credentials and API keys
Volume mounts - Static configuration files

echo -n "super-secret-password" | \
  docker secret create cassandra_password -

Health Monitoring

All services include health checks for automatic recovery:

healthcheck:
  test: ['CMD-SHELL', 'nodetool status']
  interval: 30s
  timeout: 10s
  retries: 5
  start_period: 120s

Next Steps

livekitctl

Bootstrap LiveKit infrastructure for voice and video

Cassandra

Configure and manage the Cassandra database cluster

Monitoring

Set up observability with SigNoz and OpenTelemetry

Backup & Restore

Implement backup strategies and disaster recovery

Infrastructure

​Architecture

​Core Infrastructure Components

Database

Caching & State

Message Queue

Monitoring

​Infrastructure Services

​Data Layer

​Observability Stack

​Media & Proxies

​Supporting Services

​Deployment Model

​Docker Swarm

​Resource Allocation

​Production Specifications

​Network Configuration

​Port Allocations

​Overlay Network

​Configuration Management

​Health Monitoring

​Next Steps

livekitctl

Cassandra

Monitoring

Backup & Restore

Build docs developers (and LLMs) love

Architecture

Core Infrastructure Components

Infrastructure Services

Data Layer

Observability Stack

Media & Proxies

Supporting Services

Deployment Model

Docker Swarm

Resource Allocation

Production Specifications

Network Configuration

Port Allocations

Overlay Network

Configuration Management

Health Monitoring

Next Steps