Docker Setup

Overview

Macro uses Docker Compose to run all services locally, providing a complete development environment that mirrors production infrastructure.

Architecture

The Docker setup is split across multiple compose files:

docker-compose.yml - Main application services
docker-compose-databases.yml - Database services (PostgreSQL, Redis, OpenSearch)
infra/stacks/fusionauth-instance/docker-compose.yml - FusionAuth authentication

Prerequisites

Docker and Docker Compose
Just (command runner)
Doppler (for secrets management)
AWS CLI (for credentials)
SQLX CLI (for database migrations)

Initial Setup

1. Export SOPS KMS ARN

export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"

You can skip this if using nix-shell.

2. Run Setup

just setup

This command will:

Decrypt and prepare environment variables (.env)
Create Docker networks (databases, auth)
Setup LocalStack for AWS emulation
Initialize local databases
Setup FusionAuth
Build development service images

Docker Networks

Three Docker networks connect services:

just create_networks

Creates:

databases - For database connections
auth - For FusionAuth
services - For inter-service communication

Database Services

PostgreSQL

Service: postgres
Image: postgres:16
Port: 5432
User: user
Password: password
Volume: macro_postgres_data

Health Check:

pg_isready -U postgres

Redis

Service: redis
Image: redis/redis-stack:latest
Ports: 
  - 6379 (Redis)
  - 8001 (Redis Stack UI)
Volume: macro_redis_data

Features:

Redis Stack includes RedisJSON, RedisSearch, RedisGraph
Web UI available at http://localhost:8001

OpenSearch

Service: search
Image: opensearchproject/opensearch:latest
Ports:
  - 9200 (REST API)
  - 9600 (Performance Analyzer)
Volume: macro_opensearch_data

Configuration:

Single-node cluster
Security disabled for local development
512MB JVM heap
Admin password: yourStrongPassword123!

Running Databases Only

just run_dbs

# Or detached mode
just run_dbs -d

# Stop databases
just stop-databases

Application Services

All services are built from rust/cloud-storage with different targets:

Core Services

authentication-service (Port 8080)

Dependencies: postgres, fusionauth, redis
Networks: auth, databases, services
Context: ./rust/cloud-storage
Target: authentication_service

document_storage_service (Port 8086)

Dependencies: redis, connection_gateway, authentication-service
Networks: databases, services
Target: document_storage_service

email_service (Port 8087)

Dependencies: authentication-service, document_storage_service, 
              connection_gateway, static_file_service, redis
Networks: databases, services
Target: email_service

notification_service (Port 8089)

Dependencies: redis, document_cognition_service, 
              authentication-service, connection_gateway,
              document_storage_service
Networks: databases, services
Target: notification_service

Processing Services

document_cognition_service (Port 8085)

Dependencies: document_storage_service, email_service, static_file_service
Networks: databases, services
Target: document_cognition_service

search_processing_service (Port 8092)

Profile: processors
Dependencies: email_service
Networks: databases, services
Dockerfile: Dockerfile.search_processing_service.dev

Note: Not run by default - only needed when processing search indexing.

Gateway and Connection Services

connection_gateway (Port 8082)

Dependencies: redis
Networks: databases, services
Target: connection_gateway_service

websocket_service (Port 6969)

Context: ./js/websocket-service
Networks: services

Simple WebSocket service that responds with “ping” to all messages.

File Services

static_file_service (Port 8094)

Networks: databases, services
Target: static_file_service

static_file_cdn (Port 8100)

Image: nginx:alpine
Config: ./infra/local/nginx/static-file-cdn.conf

Local CloudFront emulation - routes to S3 and static_file_service.

Utility Services

contacts_service (Port 8083)

Networks: databases, services
Target: contacts_service

unfurl_service (Port 8095)

Networks: services
Target: unfurl_service

URL unfurling with no dependencies. image_proxy_service (Port 8097)

Networks: services
Target: image_proxy_service

Proxies external images to avoid CORS issues and IP leakage.

Sync and Conversion Services

sync_service (Port 8787)

Context: ./rust/sync-service
Networks: services
Environment: INTERNAL_API_SECRET_KEY

Document sync using Cloudflare Workers/Durable Objects (via miniflare). lexical_service (Port 8096)

Context: ./js/lexical-service
Dependencies: sync_service
Environment:
  - PORT=8096
  - INTERNAL_AUTH_KEY
  - SYNC_SERVICE_AUTH_KEY
  - SYNC_SERVICE_URL=http://sync_service:8787

Converts Lexical JSON to various formats.

Running Services Locally

Start All Services

just run_local

This will:

Create Docker networks
Start all services defined in docker-compose files

Start with Rebuild

If you’ve updated code and need fresh images:

just run_local --build

Start Specific Services

# Start only databases and auth service
docker compose up postgres redis authentication-service

# Start with processors profile
docker compose --profile processors up

Stop Services

just stop-local

# Or stop everything including volumes
docker compose down -v

Health Checks

All services include health checks:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 10s

Check service health:

docker compose ps

Environment Variables

Services use a common environment configuration:

x-common-env: &common-env
  env_file:
    - .env

All services inherit from common-env anchor.

Get Environment File

# Decrypt from SOPS
just get_environment

# With AWS credential injection
just get_environment -local

# Edit encrypted environment
just edit_environment

Database Management

Initialize Databases

just setup_local_dbs

This will:

Start databases in detached mode
Create MacroDB
Run migrations
Stop databases

Individual Database Setup

just setup_macrodb      # Main database
just setup_contactsdb   # Contacts database

Docker Cache Management

Clear All Build Cache

just docker_cache_clear

Forces cold rebuild next time.

Clear Rust Target Cache Only

just docker_cache_clear_targets

Keeps downloaded crates but forces recompilation.

Check Cache Usage

just docker_cache_usage

FusionAuth Setup

Start FusionAuth

just setup_fusionauth

This will:

Create Docker networks
Start FusionAuth containers
Wait for healthy status
Run Pulumi configuration

Stop FusionAuth

just stop_fusionauth

LocalStack

AWS service emulation for local development:

just setup_localstack

Provides local versions of:

S3
SQS
Lambda
DynamoDB
Secrets Manager

Troubleshooting

Service Won’t Start

Check logs:
```
docker compose logs -f [service-name]
```
Verify dependencies are healthy:
```
docker compose ps
```
Rebuild images:
```
just run_local --build
```

Database Connection Issues

Ensure databases are running:

docker compose ps postgres redis search

Check migrations are applied:
```
just setup_macrodb
```
Verify .env file has correct credentials

Port Conflicts

If ports are already in use:

Stop conflicting services
Or modify port mappings in docker-compose.yml

Clean Reset

just destroy

This removes all containers and volumes for a fresh start.

Building Service Images

Services use multi-stage Dockerfiles:

# Build all dev service images
just rust/cloud-storage/build_dev_service_images

# Build specific service
docker compose build [service-name]

Common Workflows

Full Development Environment

# First time setup
just setup

# Start everything
just run_local

# Check all services healthy
docker compose ps

Backend Development

# Start databases and core services
docker compose up -d postgres redis search authentication-service

# Run your service locally outside Docker for faster iteration
cd rust/cloud-storage
cargo run -p my_service

Frontend Development

# Start all backend services
just run_local -d

# Run frontend
cd js/app && bun i && just local

Testing Changes

# Rebuild and restart single service
docker compose up -d --build document_storage_service

# View logs
docker compose logs -f document_storage_service

Overview

Frontend

Backend

Infrastructure

​Overview

​Architecture

​Prerequisites

​Initial Setup

​1. Export SOPS KMS ARN

​2. Run Setup

​Docker Networks

​Database Services

​PostgreSQL

​Redis

​OpenSearch

​Running Databases Only

​Application Services

​Core Services

​Processing Services

​Gateway and Connection Services

​File Services

​Utility Services

​Sync and Conversion Services

​Running Services Locally

​Start All Services

​Start with Rebuild

​Start Specific Services

​Stop Services

​Health Checks

​Environment Variables

​Get Environment File

​Database Management

​Initialize Databases

​Individual Database Setup

​Docker Cache Management

​Clear All Build Cache

​Clear Rust Target Cache Only

​Check Cache Usage

​FusionAuth Setup

​Start FusionAuth

​Stop FusionAuth

​LocalStack

​Troubleshooting

​Service Won’t Start

​Database Connection Issues

​Port Conflicts

​Clean Reset

​Building Service Images

​Common Workflows

​Full Development Environment

​Backend Development

​Frontend Development

​Testing Changes

Build docs developers (and LLMs) love

Overview

Architecture

Prerequisites

Initial Setup

1. Export SOPS KMS ARN

2. Run Setup

Docker Networks

Database Services

PostgreSQL

Redis

OpenSearch

Running Databases Only

Application Services

Core Services

Processing Services

Gateway and Connection Services

File Services

Utility Services

Sync and Conversion Services

Running Services Locally

Start All Services

Start with Rebuild

Start Specific Services

Stop Services

Health Checks

Environment Variables

Get Environment File

Database Management

Initialize Databases

Individual Database Setup

Docker Cache Management

Clear All Build Cache

Clear Rust Target Cache Only

Check Cache Usage

FusionAuth Setup

Start FusionAuth

Stop FusionAuth

LocalStack

Troubleshooting

Service Won’t Start

Database Connection Issues

Port Conflicts

Clean Reset

Building Service Images

Common Workflows

Full Development Environment

Backend Development

Frontend Development

Testing Changes