Storage Backends - Infinitic

Infinitic requires a storage backend to persist workflow and task state. This ensures reliability, enables recovery from failures, and maintains consistency across distributed workers.

Overview

Infinitic supports multiple storage backends:

Redis - Fast, in-memory data store (recommended for most use cases)
PostgreSQL - Reliable relational database
MySQL - Popular relational database
In-Memory - For development and testing only

The storage configuration is defined in your YAML configuration file under the storage section.

What Gets Stored

Infinitic persists:

Workflow state - Current workflow execution state and history
Task state - Task execution status and results
Deferred messages - Scheduled tasks and timers
Tags - Custom identifiers for workflows and tasks

Redis Storage

Redis is the recommended storage backend for production use, offering excellent performance and reliability.

Basic Configuration

storage:
  redis:
    host: localhost
    port: 6379

Complete Configuration

storage:
  redis:
    host: redis.example.com
    port: 6379
    username: infinitic
    password: ${REDIS_PASSWORD}
    database: 0
    timeout: 2000
    ssl: true
    poolConfig:
      maxTotal: 50
      maxIdle: 20
      minIdle: 5
  compression: gzip
  cache:
    keyValue:
      maximumSize: 10000
      expireAfterAccessSeconds: 3600

Redis Parameters

redis.host

string

required

Redis server hostname or IP address. Cannot be blank.Example: localhost or redis.example.com

redis.port

integer

required

Redis server port number. Must be greater than 0.Default: 6379

redis.username

string

Username for Redis authentication (Redis 6+).

redis.password

string

Password for Redis authentication.

redis.database

integer

Redis database index to use. Must be >= 0.Default: 0

redis.timeout

integer

Socket timeout in milliseconds.Default: 2000

redis.ssl

boolean

Enable SSL/TLS connection to Redis.Default: false

Connection Pool Configuration

redis.poolConfig.maxTotal

integer

Maximum number of connections in the pool. Use -1 for unlimited.Default: -1

redis.poolConfig.maxIdle

integer

Maximum number of idle connections in the pool.Default: 8

redis.poolConfig.minIdle

integer

Minimum number of idle connections to maintain.Default: 0

Configuration via Code

import io.infinitic.storage.config.RedisStorageConfig
import io.infinitic.storage.config.RedisConfig

val storage = RedisStorageConfig(
    redis = RedisConfig(
        host = "redis.example.com",
        port = 6379,
        password = System.getenv("REDIS_PASSWORD"),
        ssl = true
    )
)

PostgreSQL Storage

PostgreSQL provides ACID-compliant persistent storage with excellent reliability.

Basic Configuration

storage:
  postgres:
    host: localhost
    port: 5432
    username: infinitic
    password: ${POSTGRES_PASSWORD}

Complete Configuration

storage:
  postgres:
    host: postgres.example.com
    port: 5432
    username: infinitic
    password: ${POSTGRES_PASSWORD}
    database: infinitic
    schema: infinitic
    keySetTable: key_set_storage
    keyValueTable: key_value_storage
    maximumPoolSize: 20
    minimumIdle: 5
    idleTimeout: 600000
    connectionTimeout: 30000
    maxLifetime: 1800000
  compression: gzip

PostgreSQL Parameters

postgres.host

string

required

PostgreSQL server hostname or IP address.Example: localhost or postgres.example.com

postgres.port

integer

required

PostgreSQL server port number.Default: 5432

postgres.username

string

required

Database username for authentication.

postgres.password

string

Database password for authentication.

postgres.database

string

Database name. Must be a valid PostgreSQL database name.Default: postgres

postgres.schema

string

Schema name within the database. Created automatically if it doesn’t exist.Default: infinitic

postgres.keySetTable

string

Table name for key-set storage. Created automatically if it doesn’t exist.Default: key_set_storage

postgres.keyValueTable

string

Table name for key-value storage. Created automatically if it doesn’t exist.Default: key_value_storage

Connection Pool Parameters

postgres.maximumPoolSize

integer

Maximum number of connections in the HikariCP pool. Must be > 0.Default: HikariCP default (10)

postgres.minimumIdle

integer

Minimum number of idle connections. Must be >= 0.Default: HikariCP default (same as maximumPoolSize)

postgres.idleTimeout

integer

Maximum time (in milliseconds) a connection can sit idle in the pool. Must be > 0.Default: HikariCP default (600000ms / 10 minutes)

postgres.connectionTimeout

integer

Maximum time (in milliseconds) to wait for a connection from the pool. Must be > 0.Default: HikariCP default (30000ms / 30 seconds)

postgres.maxLifetime

integer

Maximum lifetime (in milliseconds) of a connection in the pool. Must be > 0.Default: HikariCP default (1800000ms / 30 minutes)

Database Setup

Infinitic automatically creates the database, schema, and tables if they don’t exist. Ensure the configured user has appropriate permissions:

-- Create database and user
CREATE DATABASE infinitic;
CREATE USER infinitic WITH PASSWORD 'your_password';

-- Grant permissions
GRANT ALL PRIVILEGES ON DATABASE infinitic TO infinitic;

-- Connect to database and grant schema permissions
\c infinitic
GRANT ALL ON SCHEMA infinitic TO infinitic;

Configuration via Code

import io.infinitic.storage.config.PostgresStorageConfig
import io.infinitic.storage.config.PostgresConfig

val storage = PostgresStorageConfig(
    postgres = PostgresConfig(
        host = "postgres.example.com",
        port = 5432,
        username = "infinitic",
        password = System.getenv("POSTGRES_PASSWORD"),
        database = "infinitic",
        maximumPoolSize = 20
    )
)

MySQL Storage

MySQL provides reliable persistent storage with wide adoption.

Basic Configuration

storage:
  mysql:
    host: localhost
    port: 3306
    username: infinitic
    password: ${MYSQL_PASSWORD}

Complete Configuration

storage:
  mysql:
    host: mysql.example.com
    port: 3306
    username: infinitic
    password: ${MYSQL_PASSWORD}
    database: infinitic
    keySetTable: key_set_storage
    keyValueTable: key_value_storage
    maximumPoolSize: 20
    minimumIdle: 5
    idleTimeout: 600000
    connectionTimeout: 30000
    maxLifetime: 1800000
  compression: gzip

MySQL Parameters

mysql.host

string

required

MySQL server hostname or IP address.Example: localhost or mysql.example.com

mysql.port

integer

required

MySQL server port number.Default: 3306

mysql.username

string

required

Database username for authentication.

mysql.password

string

Database password for authentication.

mysql.database

string

Database name. Must be a valid MySQL database name.Default: infinitic

mysql.keySetTable

string

Table name for key-set storage. Created automatically if it doesn’t exist.Default: key_set_storage

mysql.keyValueTable

string

Table name for key-value storage. Created automatically if it doesn’t exist.Default: key_value_storage

Connection pool parameters for MySQL are identical to PostgreSQL (see above).

Database Setup

-- Create database and user
CREATE DATABASE infinitic;
CREATE USER 'infinitic'@'%' IDENTIFIED BY 'your_password';

-- Grant permissions
GRANT ALL PRIVILEGES ON infinitic.* TO 'infinitic'@'%';
FLUSH PRIVILEGES;

Configuration via Code

import io.infinitic.storage.config.MySQLStorageConfig
import io.infinitic.storage.config.MySQLConfig

val storage = MySQLStorageConfig(
    mysql = MySQLConfig(
        host = "mysql.example.com",
        port = 3306,
        username = "infinitic",
        password = System.getenv("MYSQL_PASSWORD"),
        database = "infinitic"
    )
)

In-Memory Storage

In-memory storage is only suitable for development and testing. All data is lost when the application stops.

storage:
  inMemory:

This backend requires no configuration and stores everything in memory. Use only for local development.

Data Compression

Enable compression to reduce storage size and network bandwidth:

storage:
  redis:
    # ... redis config
  compression: gzip  # Options: gzip, bzip2, deflate

compression

string

Compression algorithm for stored data.Options:

gzip - Good balance of speed and compression ratio (recommended)
bzip2 - Better compression, slower
deflate - Similar to gzip

Default: No compression

Compression is particularly beneficial for:

Large workflow states
Workflows with significant data payloads
Reducing network transfer costs

Caching

Enable in-memory caching to reduce database load:

storage:
  redis:
    # ... redis config
  cache:
    keyValue:
      maximumSize: 10000
      expireAfterAccessSeconds: 3600
    keySet:
      maximumSize: 1000
      expireAfterAccessSeconds: 1800

Cache Parameters

cache.keyValue.maximumSize

integer

Maximum number of key-value entries to cache.Default: No caching

cache.keyValue.expireAfterAccessSeconds

integer

Seconds after last access to expire cached entries.Default: No expiration

cache.keySet.maximumSize

integer

Maximum number of key-set entries to cache.Default: No caching

cache.keySet.expireAfterAccessSeconds

integer

Seconds after last access to expire cached entries.Default: No expiration

Storage Selection Guide

Choose Redis when:

You need high performance with sub-millisecond latency
Your infrastructure already uses Redis
You want simplicity with excellent performance
Memory cost is acceptable for your data size

Choose PostgreSQL when:

You need ACID compliance and strong consistency
You prefer relational databases
You have existing PostgreSQL infrastructure
Long-term data retention is critical
You need complex querying capabilities

Choose MySQL when:

Your organization standardizes on MySQL
You have existing MySQL expertise and infrastructure
Requirements are similar to PostgreSQL

Production Best Practices

High Availability

Redis: Use Redis Cluster or Redis Sentinel for HA
PostgreSQL: Configure streaming replication with automatic failover
MySQL: Set up MySQL Group Replication or master-slave replication

Backups

Schedule regular backups of your storage backend
Test restore procedures regularly
Consider point-in-time recovery capabilities
Store backups in a different geographic region

Monitoring

Monitor these key metrics:

Connection pool utilization
Query latency (p50, p95, p99)
Storage capacity and growth rate
Error rates and failed queries
Cache hit rates (if caching enabled)

Security

Encryption at rest - Enable database encryption
Encryption in transit - Use SSL/TLS connections
Access control - Use dedicated users with minimal privileges
Network isolation - Keep storage in private network
Secrets management - Use environment variables or secret managers for passwords

Performance Tuning

For Redis:

storage:
  redis:
    host: redis.example.com
    port: 6379
    poolConfig:
      maxTotal: 100  # Increase for high concurrency
      maxIdle: 50
      minIdle: 10
  compression: gzip  # Reduce memory usage
  cache:
    keyValue:
      maximumSize: 50000  # Reduce database hits
      expireAfterAccessSeconds: 3600

For PostgreSQL/MySQL:

storage:
  postgres:  # or mysql
    host: db.example.com
    port: 5432
    username: infinitic
    password: ${DB_PASSWORD}
    maximumPoolSize: 50  # Tune based on workload
    minimumIdle: 10
    connectionTimeout: 30000
  compression: gzip  # Reduce storage size

Migration Between Backends

When migrating between storage backends:

Plan for downtime - Migration requires stopping active workflows
Export data - Use backup/export tools for your current backend
Transform data - Adapt data format if necessary
Import data - Load into new backend
Validate - Verify all data migrated correctly
Update configuration - Point workers to new storage

Storage backend migration requires careful planning. Test thoroughly in a staging environment first.

Troubleshooting

Connection Issues

Symptom: “Connection refused” or “Timeout” errors Solutions:

Verify host and port are correct
Check firewall rules allow connections
Ensure storage service is running
Verify network connectivity

Performance Issues

Symptom: Slow workflow execution or high latency Solutions:

Increase connection pool size
Enable compression to reduce data transfer
Add caching to reduce database load
Scale your storage backend (add replicas/shards)
Check database query performance

Storage Growth

Symptom: Database size growing rapidly Solutions:

Enable compression
Review data retention policies
Archive completed workflows
Monitor for workflow memory leaks

Next Steps

Pulsar Transport - Configure message transport
Deployment - Deploy to production

Getting Started

Core Concepts

Workflows

Services

Configuration

Infrastructure

Advanced

​Overview

​What Gets Stored

​Redis Storage

​Basic Configuration

​Complete Configuration

​Redis Parameters

​Connection Pool Configuration

​Configuration via Code

​PostgreSQL Storage

​Basic Configuration

​Complete Configuration

​PostgreSQL Parameters

​Connection Pool Parameters

​Database Setup

​Configuration via Code

​MySQL Storage

​Basic Configuration

​Complete Configuration

​MySQL Parameters

​Database Setup

​Configuration via Code

​In-Memory Storage

​Data Compression

​Caching

​Cache Parameters

​Storage Selection Guide

​Choose Redis when:

​Choose PostgreSQL when:

​Choose MySQL when:

​Production Best Practices

​High Availability

​Backups

​Monitoring

​Security

​Performance Tuning

​For Redis:

​For PostgreSQL/MySQL:

​Migration Between Backends

​Troubleshooting

​Connection Issues

​Performance Issues

​Storage Growth

​Next Steps

Build docs developers (and LLMs) love

Overview

What Gets Stored

Redis Storage

Basic Configuration

Complete Configuration

Redis Parameters

Connection Pool Configuration

Configuration via Code

PostgreSQL Storage

Basic Configuration

Complete Configuration

PostgreSQL Parameters

Connection Pool Parameters

Database Setup

Configuration via Code

MySQL Storage

Basic Configuration

Complete Configuration

MySQL Parameters

Database Setup

Configuration via Code

In-Memory Storage

Data Compression

Caching

Cache Parameters

Storage Selection Guide

Choose Redis when:

Choose PostgreSQL when:

Choose MySQL when:

Production Best Practices

High Availability

Backups

Monitoring

Security

Performance Tuning

For Redis:

For PostgreSQL/MySQL:

Migration Between Backends

Troubleshooting

Connection Issues

Performance Issues

Storage Growth

Next Steps