Scaling - Masar Eagle

Overview

Masar Eagle is designed as a distributed microservices architecture that can scale horizontally. This guide covers scaling strategies, performance optimization, and capacity planning.

Architecture Scalability

Service Independence

Each service can scale independently:

Stateless Services

All API services are stateless, enabling easy horizontal scaling:

No in-memory session state
JWT-based authentication (no server-side sessions)
Shared database for persistence
Message queue for asynchronous communication

Scaling Strategies

Horizontal Scaling

Gateway Layer

Scale the Gateway service to handle more concurrent requests:

# docker-compose.yml
gateway:
  image: masaregle/gateway:latest
  deploy:
    replicas: 3
    resources:
      limits:
        cpus: '1.0'
        memory: 512M

API Services

Scale individual services based on load:

user-service:
  deploy:
    replicas: 2  # Moderate traffic

trip-service:
  deploy:
    replicas: 4  # High traffic

identity-service:
  deploy:
    replicas: 2  # Auth traffic

Load Balancing

Use YARP reverse proxy with service discovery (Gateway.Api/Program.cs:39-42):

builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"))
    .AddServiceDiscoveryDestinationResolver()
    .AddTransforms(context => context.AddXForwarded(ForwardedTransformActions.Set));

Database Scaling

Read Replicas

Configure PostgreSQL read replicas for read-heavy workloads:

// Add read-only connection string
services.AddDbContext<AppDbContext>(options =>
{
    if (isReadOnlyOperation)
        options.UseNpgsql(readReplicaConnectionString);
    else
        options.UseNpgsql(primaryConnectionString);
});

Use Cases:

Trip search queries
Report generation
Analytics queries
Admin dashboard data

Connection Pooling

PostgreSQL connection pooling is enabled by default in Npgsql:

Server=postgres;Database=users;Pooling=true;MinPoolSize=5;MaxPoolSize=100;

Tuning:

MinPoolSize: Keep connections warm (5-10)
MaxPoolSize: Limit per service instance (50-100)
Monitor: npgsql_connection_pools_num_connections

Database Sharding

Consider sharding for very high scale:By Geography:

Separate databases per region/city
Users/Trips partitioned by zone

By Entity Type:

Already separated: Users, Trips, Notifications, Auth
Can further split if individual databases grow large

Indexing Strategy

Key indexes for performance:

-- Trips table
CREATE INDEX idx_trips_status ON trips(status);
CREATE INDEX idx_trips_driver_id ON trips(driver_id);
CREATE INDEX idx_trips_departure_time ON trips(departure_time);
CREATE INDEX idx_trips_zone_id ON trips(zone_id);

-- Users table
CREATE INDEX idx_users_email ON users(email);
CREATE INDEX idx_users_phone ON users(phone_number);
CREATE INDEX idx_users_role ON users(role);

-- Composite indexes
CREATE INDEX idx_trips_status_departure ON trips(status, departure_time);
CREATE INDEX idx_trips_zone_status ON trips(zone_id, status);

Message Queue Scaling

RabbitMQ Clustering

Scale RabbitMQ for high message throughput:

rabbitmq:
  image: rabbitmq:3-management
  environment:
    RABBITMQ_CLUSTER_NODES: rabbit@node1,rabbit@node2,rabbit@node3
  deploy:
    replicas: 3

Benefits:

High availability
Message distribution
Increased throughput

Queue Configuration

Optimize queue settings (AppHost.cs:21-23):

builder.AddRabbitMQ(Components.RabbitMQ, username, password)
    .WithDataVolume(Components.RabbitMQConfig.DataVolumeName)
    .WithManagementPlugin(port: int.Parse(Components.RabbitMQConfig.ManagementPort));

Production Settings:

Enable lazy queues for large queues
Set message TTL for transient messages
Configure dead letter exchanges
Use quorum queues for critical data

Wolverine Outbox

Transactional outbox ensures reliable messaging (Program.cs:116-122):

await builder.UseWolverineWithRabbitMqAsync(
    new WolverineMessagingOptions
    {
        EnablePostgresOutbox = true,
        PostgresConnectionName = Components.Database.User,
        OutboxSchema = "wolverine"
    },
    ...
);

Scaling Consideration:

Outbox polling can be CPU-intensive
Adjust polling interval based on message volume
Monitor outbox table size

Performance Optimization

Response Compression

Compression is enabled by default (Users.Api/Program.cs:41-55):

builder.Services.AddResponseCompression(options =>
{
    options.EnableForHttps = true;
    options.Providers.Add<BrotliCompressionProvider>();
    options.Providers.Add<GzipCompressionProvider>();
    options.MimeTypes = ResponseCompressionDefaults.MimeTypes.Concat(new[]
    {
        "application/json",
        "text/json"
    });
});

builder.Services.Configure<BrotliCompressionProviderOptions>(
    options => options.Level = CompressionLevel.Fastest);

Impact:

60-80% reduction in JSON response size
Lower bandwidth costs
Faster mobile client performance

Caching Strategy

HTTP Response Caching

Add response caching for static data:

app.MapGet("/api/cities", async (DbContext db) => 
    await db.Cities.ToListAsync())
    .CacheOutput(policy => policy.Expire(TimeSpan.FromHours(1)));

Distributed Cache

Use Redis for shared cache:

builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "redis:6379";
    options.InstanceName = "MasarEagle_";
});

In-Memory Caching

Cache configuration data:

services.AddMemoryCache();

cache.GetOrCreate("AppSettings", entry =>
{
    entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(15);
    return LoadAppSettings();
});

EF Core Query Caching

Queries are automatically cached:

// Compiled query - cached once, reused
var query = EF.CompileQuery(
    (DbContext db, int id) => db.Users.Find(id)
);

Background Jobs

Hangfire is configured for background processing (Trips.Api/Program.cs:115-127):

builder.Services.AddHangfire(config => config
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseMemoryStorage());

builder.Services.AddHangfireServer(options =>
{
    options.Queues = ["default", "notifications"];
    options.WorkerCount = Environment.ProcessorCount * 5;
});

Memory Storage: Current configuration uses in-memory storage. For production:

Use PostgreSQL storage: UsePostgreSqlStorage()
Prevents job loss on restart
Enables distributed job execution

Job Examples:

Trip reminders (TripReminder cron: */5 * * * *)
Auto-cancel overdue trips (AutoCancel cron: */10 * * * *)
Account purge (AccountPurgeWorker)

Request Size Limits

Large file uploads are supported (Users.Api/Program.cs:26-33):

builder.WebHost.ConfigureKestrel(options => 
    options.Limits.MaxRequestBodySize = null);

builder.Services.Configure<FormOptions>(options =>
{
    options.MultipartBodyLengthLimit = long.MaxValue;
    options.ValueLengthLimit = int.MaxValue;
    options.ValueCountLimit = int.MaxValue; 
});

Production Recommendations:

Set reasonable limits (e.g., 10MB for images)
Use streaming for large files
Offload to object storage (S3, Azure Blob)
Implement chunked uploads for >5MB

Monitoring for Scale

Key Metrics

Service Metrics
Database Metrics
Queue Metrics
Runtime Metrics

# Request rate
rate(http_server_request_duration_seconds_count[5m])

# Error rate
rate(errors_total[5m])

# Latency (p95)
histogram_quantile(0.95, rate(http_server_request_duration_seconds_bucket[5m]))

# Active requests
http_server_active_requests

# Connection pool usage
npgsql_connection_pools_num_connections

# Active connections
pg_stat_activity_count

# Query duration
pg_stat_statements_mean_time_seconds

# Database size
pg_database_size_bytes

# Queue depth
rabbitmq_queue_messages

# Message rate
rate(rabbitmq_queue_messages_published_total[5m])

# Consumer count
rabbitmq_queue_consumers

# Unacked messages
rabbitmq_queue_messages_unacked

# Memory
process_runtime_dotnet_gc_heap_size_bytes

# CPU
process_cpu_seconds_total

# Thread pool
process_runtime_dotnet_thread_pool_threads_count

# GC collections
process_runtime_dotnet_gc_collections_count

SLO Targets

Metric	Target	Critical Threshold
Availability	Greater than 99.9%	Less than 99%
P95 Latency	Less than 500ms	Greater than 2s
Error Rate	Less than 1%	Greater than 5%
Database Conn Pool	Less than 80%	Greater than 95%
Queue Lag	Less than 100 msgs	Greater than 1000 msgs

Capacity Planning

Resource Estimates

Per Service Instance

Minimum Resources:

CPU: 0.5 cores
Memory: 256MB
Disk: 100MB

Recommended (Production):

CPU: 1-2 cores
Memory: 512MB-1GB
Disk: 1GB (for logs)

Capacity:

~200-500 req/sec per instance
Scales linearly with CPU

PostgreSQL

Sizing:

connections = (num_cpu_cores * 2) + effective_spindle_count
shared_buffers = 25% of RAM
effective_cache_size = 50% of RAM
work_mem = (RAM - shared_buffers) / max_connections

Example (16GB RAM, 4 cores):

max_connections: 200
shared_buffers: 4GB
effective_cache_size: 8GB
work_mem: 60MB

RabbitMQ

Minimum:

CPU: 1 core
Memory: 512MB
Disk: 10GB

Production:

CPU: 2-4 cores
Memory: 2-4GB
Disk: 50GB+ (message persistence)

Capacity:

~10,000 messages/sec
50,000+ concurrent connections

Growth Projections

Baseline Metrics

Collect 1-2 weeks of production metrics:

Average requests/sec
Peak requests/sec
Database query count
Message queue throughput

Calculate Growth

Project growth based on business metrics:

Future Load = Current Load * (1 + Growth Rate)^Time

Example:
- Current: 1,000 trips/day
- Growth: 20% monthly
- 6 months: 1,000 * 1.2^6 = 2,986 trips/day

Add Headroom

Add 50-100% headroom for:

Traffic spikes
Failover capacity
Deployment safety

Plan Scaling Events

Schedule scaling before limits:

Database: When >70% CPU or connections
Services: When >60% CPU or memory
Queue: When lag >50% of threshold

Deployment Strategies

Blue-Green Deployment

Rolling Updates

deploy:
  update_config:
    parallelism: 1  # Update 1 at a time
    delay: 30s      # Wait between updates
    failure_action: rollback
    monitor: 60s    # Monitor after update

Best Practices

Load Test Before Scaling

Use tools like k6, JMeter, or Artillery to simulate production load:

// k6 load test
export default function() {
  http.get('http://gateway/api/trips');
}

export let options = {
  vus: 100,  // 100 virtual users
  duration: '5m'
};

Monitor Everything

Use the observability stack:

Metrics: Prometheus + Grafana
Logs: Loki
Traces: Jaeger
Alerts: Prometheus Alertmanager

Implement Circuit Breakers

Protect against cascading failures:

services.AddHttpClient("trip")
    .AddStandardResilienceHandler();

Already configured via ServiceDefaults!

Use Async All The Way

Never block threads:

// Good
var result = await service.GetDataAsync();

// Bad - blocks thread!
var result = service.GetDataAsync().Result;

Getting Started

Setup & Deployment

Core Concepts

Services

Features

Development

Operations

​Overview

​Architecture Scalability

​Service Independence

​Stateless Services

​Scaling Strategies

​Horizontal Scaling

​Database Scaling

​Message Queue Scaling

​Performance Optimization

​Response Compression

​Caching Strategy

HTTP Response Caching

Distributed Cache

In-Memory Caching

EF Core Query Caching

​Background Jobs

​Request Size Limits

​Monitoring for Scale

​Key Metrics

​SLO Targets

​Capacity Planning

​Resource Estimates

​Growth Projections

​Deployment Strategies

​Blue-Green Deployment

​Rolling Updates

​Best Practices

Load Test Before Scaling

Monitor Everything

Implement Circuit Breakers

Use Async All The Way

​Next Steps

Monitoring

Troubleshooting

Build docs developers (and LLMs) love

Overview

Architecture Scalability

Service Independence

Stateless Services

Scaling Strategies

Horizontal Scaling

Database Scaling

Message Queue Scaling

Performance Optimization

Response Compression

Caching Strategy

Background Jobs

Request Size Limits

Monitoring for Scale

Key Metrics

SLO Targets

Capacity Planning

Resource Estimates

Growth Projections

Deployment Strategies

Blue-Green Deployment

Rolling Updates

Best Practices

Next Steps