Performance Optimization

Blnk is designed for high-performance financial operations. This guide covers database optimization, caching strategies, batch processing, and monitoring to achieve optimal throughput.

Database Optimization

Connection Pooling

PostgreSQL connection pool configuration is critical for performance:

{
  "data_source": {
    "dns": "postgresql://user:pass@localhost:5432/blnk",
    "max_open_conns": 25,
    "max_idle_conns": 10,
    "conn_max_lifetime": "30m",
    "conn_max_idle_time": "5m"
  }
}

Implementation (database/db.go:87-90):

func ConnectDB(dsConfig config.DataSourceConfig) (*sql.DB, error) {
    return pgconn.ConnectDB(dsConfig)
}

Sizing Guidelines:

Workload	Max Open	Max Idle	Rationale
Low (< 100 TPS)	10	5	Minimize resource usage
Medium (100-1000 TPS)	25	10	Default configuration
High (1000-5000 TPS)	50	20	Support burst traffic
Very High (> 5000 TPS)	100	40	Maximum parallelism

Formula:

max_open_conns = (number_of_queues * workers_per_queue) + 20
max_idle_conns = max_open_conns * 0.4

Connection Lifetime:

conn_max_lifetime: 30m (prevents stale connections)
conn_max_idle_time: 5m (releases idle connections faster)

Index Optimization

Blnk creates optimized indexes on frequently queried fields: Critical Indexes:

-- Transactions
CREATE INDEX idx_transactions_reference ON blnk.transactions(reference);
CREATE INDEX idx_transactions_source ON blnk.transactions(source);
CREATE INDEX idx_transactions_destination ON blnk.transactions(destination);
CREATE INDEX idx_transactions_created_at ON blnk.transactions(created_at DESC);
CREATE INDEX idx_transactions_status ON blnk.transactions(status);

-- Balances
CREATE INDEX idx_balances_ledger_id ON blnk.balances(ledger_id);
CREATE INDEX idx_balances_identity_id ON blnk.balances(identity_id);
CREATE INDEX idx_balances_indicator ON blnk.balances(indicator);
CREATE UNIQUE INDEX idx_balances_indicator_unique ON blnk.balances(indicator) WHERE indicator IS NOT NULL;

-- Lineage
CREATE INDEX idx_lineage_mappings_balance_id ON blnk.lineage_mappings(balance_id);
CREATE INDEX idx_lineage_mappings_provider ON blnk.lineage_mappings(balance_id, provider);

Composite Indexes for Common Queries:

-- Find transactions by source and status
CREATE INDEX idx_txn_source_status ON blnk.transactions(source, status, created_at DESC);

-- Find transactions by date range and currency
CREATE INDEX idx_txn_date_currency ON blnk.transactions(created_at DESC, currency);

Monitor Index Usage:

-- Check unused indexes
SELECT 
    schemaname,
    tablename,
    indexname,
    idx_scan,
    idx_tup_read,
    idx_tup_fetch
FROM pg_stat_user_indexes
WHERE schemaname = 'blnk'
ORDER BY idx_scan ASC;

Query Optimization

Use EXPLAIN ANALYZE:

EXPLAIN ANALYZE
SELECT * FROM blnk.transactions
WHERE source = 'bal_abc123'
AND status = 'APPLIED'
ORDER BY created_at DESC
LIMIT 100;

Look for:

Sequential scans (should use indexes)
High cost estimates
Slow execution time

Batch Queries: Instead of N individual queries:

// Bad: N queries
for _, id := range balanceIDs {
    balance, _ := datasource.GetBalanceByID(id)
    balances = append(balances, balance)
}

Use batch fetch:

// Good: 1 query
balances, _ := datasource.GetBalancesByIDsLite(ctx, balanceIDs)

Example from Lineage (lineage.go:849):

func (l *Blnk) getLineageSources(ctx context.Context, mappings []model.LineageMapping) ([]LineageSource, error) {
    // Collect all shadow balance IDs for batch query
    shadowBalanceIDs := make([]string, 0, len(mappings))
    for _, mapping := range mappings {
        shadowBalanceIDs = append(shadowBalanceIDs, mapping.ShadowBalanceID)
    }
    
    // Fetch all shadow balances in a single query
    balances, err := l.datasource.GetBalancesByIDsLite(ctx, shadowBalanceIDs)
    if err != nil {
        return nil, fmt.Errorf("failed to fetch shadow balances: %w", err)
    }
    // ...
}

PostgreSQL Configuration

Recommended postgresql.conf settings for production:

# Memory Settings
shared_buffers = 4GB                  # 25% of system RAM
effective_cache_size = 12GB           # 75% of system RAM
maintenance_work_mem = 1GB
work_mem = 32MB

# Checkpoint Settings
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100

# Parallelism
max_worker_processes = 8
max_parallel_workers_per_gather = 4
max_parallel_workers = 8

# Connections
max_connections = 200

# Logging
log_min_duration_statement = 1000     # Log queries > 1s
log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h '

For High Write Workloads:

# WAL Configuration
wal_level = replica
max_wal_size = 4GB
min_wal_size = 1GB
wal_compression = on

# Commit Delay (batch commits)
commit_delay = 100                    # microseconds
commit_siblings = 5

Redis Caching

Blnk uses a two-tier caching strategy with Redis and in-memory LRU.

Cache Architecture

Implementation (internal/cache/cache.go:84-114):

func NewCacheWithClient(client redis.UniversalClient) Cache {
    c := cache.New(&cache.Options{
        Redis:      client,
        LocalCache: cache.NewTinyLFU(cacheSize, 1*time.Minute),
    })
    return &RedisCache{cache: c}
}

const cacheSize = 128000  // Local cache size

Two-Tier Strategy:

L1 Cache (Local): TinyLFU in-memory cache (128K entries, 1-minute TTL)
L2 Cache (Redis): Distributed Redis cache (configurable TTL)

Benefits:

Sub-millisecond L1 cache hits
Shared L2 cache across instances
Automatic eviction (LFU policy)

Cache Configuration

{
  "redis": {
    "dns": "redis://localhost:6379",
    "pool_size": 100,
    "min_idle_conns": 20,
    "skip_tls_verify": false
  }
}

Pool Sizing:

pool_size = max_open_db_conns + number_of_queues + 20

Example:

Database connections: 50
Transaction queues: 20
Overhead: 20
Total pool_size: 90 (round to 100)

Caching Strategy

Balance Caching: Balances are cached with short TTL to ensure consistency:

// Cache balance for 30 seconds
cache.Set(ctx, balanceKey, balance, 30*time.Second)

Identity Caching: Identities are cached longer (less frequent changes):

// Cache identity for 5 minutes
cache.Set(ctx, identityKey, identity, 5*time.Minute)

Cache Invalidation: Caches are invalidated on updates:

// After updating balance
cache.Delete(ctx, fmt.Sprintf("balance:%s", balanceID))

Redis Memory Optimization

Set Eviction Policy:

maxmemory 2gb
maxmemory-policy allkeys-lfu  # Evict least frequently used

Monitor Memory Usage:

redis-cli INFO memory

Look for:

used_memory_human: Current memory usage
used_memory_peak_human: Peak usage
mem_fragmentation_ratio: Should be close to 1.0

Defragment if needed:

redis-cli MEMORY PURGE

Batch Processing

Transaction Batching

Process multiple transactions in parallel:

var wg sync.WaitGroup
sem := make(chan struct{}, maxWorkers)

for _, txn := range transactions {
    sem <- struct{}{}  // Acquire semaphore
    wg.Add(1)
    
    go func(t *model.Transaction) {
        defer wg.Done()
        defer func() { <-sem }()  // Release semaphore
        
        _, err := blnk.RecordTransaction(ctx, t)
        if err != nil {
            logrus.Errorf("Failed to process transaction: %v", err)
        }
    }(txn)
}

wg.Wait()

Configuration (config/config.go:41-48):

defaultTransaction = TransactionConfig{
    BatchSize:          100000,
    MaxQueueSize:       1000,
    MaxWorkers:         10,
    LockDuration:       30 * time.Minute,
    IndexQueuePrefix:   "transactions",
    EnableQueuedChecks: false,
}

Tuning:

MaxWorkers: Number of concurrent transaction processors
BatchSize: Batch size for bulk operations
MaxQueueSize: In-memory queue size

Reindex Batching

Reindexing uses configurable batch sizes:

POST /admin/reindex
{
  "batch_size": 5000
}

Optimal Batch Sizes:

1,000: Default, balanced performance
5,000: High throughput, more memory
10,000: Maximum speed, requires 8GB+ RAM

Implementation (reindex.go:54-58):

func NewReindexService(client *TypesenseClient, datasource database.IDataSource, config ReindexConfig) *ReindexService {
    if config.BatchSize <= 0 {
        config.BatchSize = 1000
    }
    // ...
}

Monitoring and Metrics

Database Metrics

Connection Pool Stats:

SELECT 
    datname,
    numbackends,
    xact_commit,
    xact_rollback,
    blks_read,
    blks_hit,
    tup_returned,
    tup_fetched
FROM pg_stat_database
WHERE datname = 'blnk';

Key Metrics:

numbackends: Active connections (should be < max_connections)
blks_hit / (blks_hit + blks_read): Cache hit ratio (target: > 0.99)
xact_commit / (xact_commit + xact_rollback): Commit ratio (target: > 0.95)

Slow Queries:

SELECT 
    query,
    calls,
    total_time,
    mean_time,
    max_time
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;

Enable pg_stat_statements:

CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

Queue Metrics

Monitor Asynq queues:

# Access Asynq dashboard
open http://localhost:5004

Metrics to Watch:

Queue depth (should be < 10,000)
Processing latency (p99 < 1s)
Failure rate (< 1%)
Retry rate

CLI Inspection:

# Queue stats
redis-cli LLEN blnk:queue:new:transaction_1

# Pending tasks
redis-cli ZCARD blnk:queue:new:transaction_1:pending

# Dead letter queue
redis-cli ZCARD blnk:queue:new:transaction_1:dead

Application Metrics

Blnk exposes Prometheus-compatible metrics:

curl http://localhost:5001/metrics

Key Metrics:

blnk_transactions_total: Total transactions processed
blnk_transaction_duration_seconds: Transaction processing time
blnk_balance_operations_total: Balance operation count
blnk_cache_hits_total: Cache hit count
blnk_cache_misses_total: Cache miss count

Grafana Dashboard:

{
  "panels": [
    {
      "title": "Transaction Throughput",
      "targets": [
        {
          "expr": "rate(blnk_transactions_total[5m])"
        }
      ]
    },
    {
      "title": "p99 Latency",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, rate(blnk_transaction_duration_seconds_bucket[5m]))"
        }
      ]
    }
  ]
}

Performance Benchmarks

Hardware Specifications

Test Environment:

CPU: 8 cores (Intel Xeon or equivalent)
RAM: 16GB
Disk: SSD (NVMe recommended)
Network: 1Gbps

Throughput

Simple Transactions (A → B):

Sequential: ~500 TPS
Queued (20 queues): ~5,000 TPS
Queued (50 queues): ~12,000 TPS

Complex Transactions (with lineage):

Sequential: ~200 TPS
Queued (20 queues): ~2,000 TPS
Queued (50 queues): ~5,000 TPS

Batch Operations:

Bulk insert (1,000 txns): ~2 seconds
Reindex (1M records): ~10 minutes

Latency

Transaction Processing:

p50: 10ms
p95: 50ms
p99: 100ms

Balance Queries:

Cache hit: < 1ms
Cache miss: 5-10ms

Search Queries:

Simple filter: 10-20ms
Complex query: 50-100ms
Full-text search: 100-200ms

Scaling Strategies

Vertical Scaling

PostgreSQL:

Increase shared_buffers (25% of RAM)
Add CPU cores for parallelism
Use faster disks (NVMe SSD)
Increase connection pool

Redis:

Increase maxmemory
Use Redis Cluster for distribution
Enable persistence (RDB + AOF)

Horizontal Scaling

Application Tier:

Run multiple Blnk instances
Use load balancer (nginx, HAProxy)
Share Redis and PostgreSQL

Database Tier:

PostgreSQL read replicas
Connection pooling (PgBouncer)
Partitioning for large tables

Queue Tier:

Increase number_of_queues
Redis Cluster for queue distribution
Multiple Asynq workers

Partitioning

Partition transactions by date:

CREATE TABLE blnk.transactions (
    transaction_id TEXT PRIMARY KEY,
    created_at TIMESTAMP NOT NULL,
    -- other fields
) PARTITION BY RANGE (created_at);

CREATE TABLE blnk.transactions_2024_01 PARTITION OF blnk.transactions
    FOR VALUES FROM ('2024-01-01') TO ('2024-02-01');

CREATE TABLE blnk.transactions_2024_02 PARTITION OF blnk.transactions
    FOR VALUES FROM ('2024-02-01') TO ('2024-03-01');

Benefits:

Faster queries (partition pruning)
Easier archival (drop old partitions)
Improved vacuum performance

Best Practices

Monitor Continuously: Set up alerts for key metrics
- Database connection pool exhaustion
- Queue depth > 10,000
- Cache hit ratio < 95%
- Transaction failure rate > 1%
Use Connection Pooling: Always configure proper pool sizes
Enable Caching: Leverage Redis cache for hot data
Batch Operations: Group related operations when possible
Index Strategically: Add indexes for query patterns, not every column

Regular Maintenance:

# Weekly
VACUUM ANALYZE blnk.transactions;
REINDEX DATABASE blnk;

Archive Old Data: Move historical data to archive tables

-- Archive transactions older than 1 year
INSERT INTO blnk.transactions_archive
SELECT * FROM blnk.transactions
WHERE created_at < NOW() - INTERVAL '1 year';

DELETE FROM blnk.transactions
WHERE created_at < NOW() - INTERVAL '1 year';

Load Testing: Regular load tests to validate configuration

# Using k6
k6 run --vus 100 --duration 5m load_test.js

Troubleshooting

High Database CPU

Symptoms:

PostgreSQL CPU > 80%
Slow query performance

Solutions:

Check for missing indexes (see Index Optimization)
Review slow query log
Optimize queries with EXPLAIN ANALYZE
Increase work_mem for complex queries
Add read replicas

Connection Pool Exhaustion

Symptoms:

Error: too many clients already

Solutions:

Increase max_connections in PostgreSQL
Increase max_open_conns in Blnk
Use connection pooler (PgBouncer)
Check for connection leaks

Redis Memory Issues

Symptoms:

High memory usage
OOM errors
Slow cache performance

Solutions:

Increase maxmemory
Enable eviction policy
Reduce cache TTL
Check for memory leaks
Use Redis Cluster

High Queue Latency

Symptoms:

Transactions taking minutes to process
Growing queue depth

Solutions:

Increase number_of_queues
Increase max_workers
Check for slow transactions (enable query logging)
Scale horizontally (more workers)
Optimize transaction logic

Getting Started

Core Concepts

Guides

Advanced

Deployment

Performance Optimization

Database Optimization

Connection Pooling

Index Optimization

Query Optimization

PostgreSQL Configuration

Redis Caching

Cache Architecture

Cache Configuration

Caching Strategy

Redis Memory Optimization

Batch Processing

Transaction Batching

Reindex Batching

Monitoring and Metrics

Database Metrics

Queue Metrics

Application Metrics

Performance Benchmarks

Hardware Specifications

Throughput

Latency

Scaling Strategies

Vertical Scaling

Horizontal Scaling

Partitioning

Best Practices

Troubleshooting

High Database CPU

Connection Pool Exhaustion

Redis Memory Issues

High Queue Latency

Build docs developers (and LLMs) love

Getting Started

Core Concepts

Guides

Advanced

Deployment

​Database Optimization

​Connection Pooling

​Index Optimization

​Query Optimization

​PostgreSQL Configuration

​Redis Caching

​Cache Architecture

​Cache Configuration

​Caching Strategy

​Redis Memory Optimization

​Batch Processing

​Transaction Batching

​Reindex Batching

​Monitoring and Metrics

​Database Metrics

​Queue Metrics

​Application Metrics

​Performance Benchmarks

​Hardware Specifications

​Throughput

​Latency

​Scaling Strategies

​Vertical Scaling

​Horizontal Scaling

​Partitioning

​Best Practices

​Troubleshooting

​High Database CPU

​Connection Pool Exhaustion

​Redis Memory Issues

​High Queue Latency

​Related Resources

Build docs developers (and LLMs) love

Database Optimization

Connection Pooling

Index Optimization

Query Optimization

PostgreSQL Configuration

Redis Caching

Cache Architecture

Cache Configuration

Caching Strategy

Redis Memory Optimization

Batch Processing

Transaction Batching

Reindex Batching

Monitoring and Metrics

Database Metrics

Queue Metrics

Application Metrics

Performance Benchmarks

Hardware Specifications

Throughput

Latency

Scaling Strategies

Vertical Scaling

Horizontal Scaling

Partitioning

Best Practices

Troubleshooting

High Database CPU

Connection Pool Exhaustion

Redis Memory Issues

High Queue Latency

Related Resources