Delta Compression

Overview

Delta compression reduces bandwidth usage by sending only the differences between consecutive messages instead of full messages. With conflation keys, messages are grouped by entity (e.g., stock symbol, device ID) for dramatically improved compression efficiency when broadcasting updates for multiple entities on a single channel. Bandwidth Savings: 60-90% typical, 80-95% with conflation keys

Quick Start

Server Configuration

config/config.json

{
  "delta_compression": {
    "enabled": true,
    "algorithm": "Fossil",
    "full_message_interval": 10,
    "min_message_size": 100,
    "max_state_age_secs": 300,
    "max_channel_states_per_socket": 100,
    "cluster_coordination": false,
    "omit_delta_algorithm": false
  },
  "apps": [{
    "id": "my-app",
    "channel_delta_compression": {
      "market-*": {
        "enabled": true,
        "algorithm": "Fossil",
        "conflation_key": "asset",
        "max_messages_per_key": 100,
        "max_conflation_keys": 1000
      },
      "notifications": "disabled"
    }
  }]
}

Client Implementation

const pusher = new Pusher('app-key', { cluster: 'mt1' });

// Enable delta compression globally
pusher.connection.bind('connected', () => {
  pusher.connection.send_event('pusher:enable_delta_compression', {});
});

// Subscribe to channel
const channel = pusher.subscribe('market-data');

// Handle cache sync (initialize local state)
channel.bind('pusher:delta_cache_sync', (data) => {
  initializeCache(channel.name, data);
});

// Handle regular events (delta reconstruction is automatic)
channel.bind('price-update', (data) => {
  console.log('Price:', data.asset, data.price);
});

Key Features

Core Capabilities

Backwards Compatible - Standard Pusher clients work without changes
Opt-in - Clients explicitly enable delta compression
Per-Socket State - Independent delta state per connection
Per-Channel Tracking - Separate state for each subscribed channel
Thread-Safe - Lock-free concurrent access (DashMap)
Automatic Fallback - Sends full messages when deltas are larger
Encrypted Channel Detection - Automatically skips private-encrypted-* channels

Conflation Keys

Entity Grouping - Messages grouped by entity (e.g., “BTC”, “ETH”)
Per-Key Delta State - Each entity maintains independent delta history
Cache Synchronization - Clients receive initial state on subscription
FIFO Eviction - Configurable cache size with automatic cleanup
Pattern Matching - Wildcard support ("market-*", "*-data")
Per-Channel Configuration - Different settings per channel

Publisher Control

Per-Publish Delta Flag - Publishers can force delta or full messages per-event
Channel-Level Configuration - Different compression settings per channel pattern

Client Negotiation

Per-Subscription Delta Settings - Clients can negotiate delta per-channel
Algorithm Selection - Clients can request specific algorithms per-channel
Flexible Formats - Support for simple string, boolean, or object configuration

Compression Algorithms

Fossil Delta (Default)

Best for: General purpose, text-heavy messages

Metric	Rating	Notes
Speed	⭐⭐⭐⭐⭐	500ns - 2µs per message
Compression	⭐⭐⭐⭐	70-85% bandwidth savings
Small messages	⭐⭐⭐⭐⭐	Excellent for < 1KB
Large messages	⭐⭐⭐	Degrades above 5KB

Xdelta3

Best for: Maximum compression, large messages

Metric	Rating	Notes
Speed	⭐⭐⭐⭐	1-2µs per message
Compression	⭐⭐⭐⭐⭐	80-90% bandwidth savings
Small messages	⭐⭐⭐	Slower than Fossil
Large messages	⭐⭐⭐⭐⭐	4x faster than Fossil at 5KB
Standard	✅	VCDIFF (RFC 3284)

Algorithm Selection

// Global default
{
  "delta_compression": {
    "algorithm": "Fossil"
  }
}

// Per-channel override
{
  "apps": [{
    "channel_delta_compression": {
      "market-*": "Fossil",
      "documents-*": "Xdelta3"
    }
  }]
}

Why Conflation Keys?

The Problem (Without Conflation)

When multiple entities share a channel, sequential comparison produces poor compression:

Channel: "market-data"
Message 1: {"asset":"BTC", "price":"100.00"}  → Full (98 bytes)
Message 2: {"asset":"ETH", "price":"1.00"}    → Delta vs BTC (59 bytes) ❌
Message 3: {"asset":"BTC", "price":"100.01"}  → Delta vs ETH (58 bytes) ❌
Message 4: {"asset":"ETH", "price":"1.01"}    → Delta vs BTC (53 bytes) ❌

Total: 268 bytes (32.6% savings)

The Solution (With Conflation Keys)

Grouping messages by entity dramatically improves compression:

Channel: "market-data", Conflation Key: "asset"
Message 1: {"asset":"BTC", "price":"100.00"}  → Full [BTC] (98 bytes)
Message 2: {"asset":"ETH", "price":"1.00"}    → Full [ETH] (95 bytes)
Message 3: {"asset":"BTC", "price":"100.01"}  → Delta [BTC] vs #1 (28 bytes) ✅
Message 4: {"asset":"ETH", "price":"1.01"}    → Delta [ETH] vs #2 (41 bytes) ✅

Total: 262 bytes (40.1% savings) → +7.5% improvement!

Real-World Impact

Scenario	Without Conflation	With Conflation	Improvement
5 messages, 2 assets	32.6% savings	40.1% savings	+7.5%
100 messages, 10 assets	~50% savings	~75% savings	+25%
1000 messages, 100 assets	~50% savings (500KB)	~85% savings (145KB)	+35%

Configuration

Global Settings

{
  "delta_compression": {
    "enabled": true,
    "algorithm": "Fossil",
    "full_message_interval": 10,
    "min_message_size": 100,
    "max_state_age_secs": 300,
    "max_channel_states_per_socket": 100
  }
}

Option	Default	Description
`enabled`	`true`	Enable/disable globally
`algorithm`	`"Fossil"`	Compression algorithm (`"Fossil"` or `"Xdelta3"`)
`full_message_interval`	`10`	Send full message every N deltas
`min_message_size`	`100`	Minimum size (bytes) to compress
`max_state_age_secs`	`300`	Max state age before cleanup
`max_channel_states_per_socket`	`100`	Max channels per socket

Per-Channel Configuration

Simple Format (String-based)

{
  "apps": [{
    "channel_delta_compression": {
      "market-*": "Fossil",
      "chat-*": "Xdelta3",
      "notifications": "disabled",
      "*": "inherit"
    }
  }]
}

Options: "Fossil", "Xdelta3", "disabled", "inherit"

Full Format (With Conflation Keys)

{
  "apps": [{
    "channel_delta_compression": {
      "ticker:*": {
        "enabled": true,
        "algorithm": "Fossil",
        "conflation_key": "item_id",
        "max_messages_per_key": 10,
        "max_conflation_keys": 1000,
        "enable_tags": false
      },
      "iot-sensors-*": {
        "enabled": true,
        "algorithm": "Xdelta3",
        "conflation_key": "device_id",
        "max_messages_per_key": 50,
        "max_conflation_keys": 500
      }
    }
  }]
}

Option	Description	Example
`enabled`	Enable for this channel	`true`
`algorithm`	Algorithm to use	`"Fossil"`, `"Xdelta3"`
`conflation_key`	JSON path to entity key	`"asset"`, `"data.symbol"`
`max_messages_per_key`	Cache size per entity	`100`
`max_conflation_keys`	Max entities to track	`1000`
`enable_tags`	Include tags in messages	`true`

Pattern Matching

Supports wildcard patterns with priority-based matching:

Pattern	Matches	Priority
`"market-data"`	Exact: `"market-data"`	1 (highest)
`"market-*"`	Prefix: `"market-btc"`, `"market-eth"`	2
`"*-data"`	Suffix: `"market-data"`, `"sensor-data"`	2
`"*"`	All channels	3 (lowest)

Priority: Exact match → First wildcard match → Server default

Per-Publish Delta Control

Publishers can control delta compression on a per-message basis:

# Force delta compression
curl -X POST http://localhost:6001/apps/my-app/events \
  -d '{
    "name": "price-update",
    "channel": "ticker:BTC",
    "data": "{\"price\": 50000}",
    "delta": true
  }'

# Force full message (skip delta)
curl -X POST http://localhost:6001/apps/my-app/events \
  -d '{
    "name": "snapshot",
    "channel": "ticker:BTC",
    "data": "{\"full_state\": {...}}",
    "delta": false
  }'

When to Use

Scenario	`delta` value	Reason
Incremental updates	`true` or omit	Small changes benefit from delta compression
Full state snapshots	`false`	Large state changes won’t compress well
Critical messages	`false`	Ensure clients get full data
Schema changes	`false`	Breaking changes need full message

Per-Subscription Delta Negotiation

Clients can negotiate delta compression settings on a per-channel basis:

// Method 1: Simple algorithm selection
const channel = pusher.subscribe('ticker:BTC', {
  delta: { enabled: true, algorithm: 'Fossil' }
});

// Method 2: Disable delta for specific channel
const snapshotChannel = pusher.subscribe('snapshots', {
  delta: { enabled: false }
});

// Method 3: Use Xdelta3 algorithm
const highCompChannel = pusher.subscribe('large-data', {
  delta: { algorithm: 'Xdelta3' }
});

// Method 4: Combined with tag filtering
const filteredChannel = pusher.subscribe('events', {
  filter: Filter.eq('type', 'important'),
  delta: { enabled: true, algorithm: 'Fossil' }
});

Subscription Message Formats

// String format
{ "channel": "ticker:BTC", "delta": "Fossil" }
{ "channel": "ticker:BTC", "delta": "disabled" }

// Boolean format
{ "channel": "ticker:BTC", "delta": true }   // Enable with default
{ "channel": "ticker:BTC", "delta": false }  // Disable

// Object format (full control)
{ "channel": "ticker:BTC", "delta": { "enabled": true, "algorithm": "Fossil" } }

Priority Order

Delta compression settings are resolved in this order (highest priority first):

Encrypted channel detection - Always disabled for private-encrypted-*
Per-subscription settings - Client’s subscription-time request
Per-channel server config - channel_delta_compression in app config
Global socket state - pusher:enable_delta_compression event
Server default - delta_compression.enabled in server config

Horizontal Scaling

Node-Local Intervals (Default)

By default, each node tracks delta intervals independently:

{
  "delta_compression": {
    "cluster_coordination": false
  }
}

Benefits:

Zero coordination overhead
No additional latency
Works with any adapter
Simpler architecture

Trade-offs:

Full messages may be sent more frequently (still ~60-90% savings)
Each node resets intervals independently

Cluster Coordination (Optional)

Enable synchronized full message intervals across all nodes:

{
  "delta_compression": {
    "cluster_coordination": true
  },
  "adapter": {
    "driver": "redis"  // or "redis-cluster" or "nats"
  }
}

Benefits:

Synchronized full message intervals cluster-wide
Maximum compression efficiency
Optimal for high-throughput scenarios

Trade-offs:

Adds ~0.5-1.2ms latency per message
Requires Redis or NATS adapter

Use Cases

✅ Perfect For

1. Market Data Feeds

Channel: "market-data", Key: "asset"
Groups: BTC, ETH, ADA, etc.
Savings: 80-90%

2. IoT Sensor Networks

Channel: "sensors", Key: "device_id"
Groups: sensor-1, sensor-2, etc.
Savings: 70-85%

3. Real-Time Gaming

Channel: "game-lobby", Key: "player_id"
Groups: player1, player2, etc.
Savings: 60-80%

4. User Presence

Channel: "presence", Key: "user_id"
Groups: user-123, user-456, etc.
Savings: 65-75%

❌ Not Beneficial

Only one entity per channel (use proper channel design)
Completely random data with no recurring patterns
Very few messages per entity (overhead not worth it)
Messages < 100 bytes (overhead exceeds savings)

Performance

Bandwidth Savings

Scenario	Compression	Typical Savings
Similar sequential messages	Yes	60-90%
Interleaved entities (no conflation)	Yes	30-50%
Interleaved entities (with conflation)	Yes	80-95%
Completely different messages	No (fallback)	0%

CPU Overhead

Operation	Time	Impact
Fossil delta generation	500ns - 2µs	Minimal
Xdelta3 delta generation	1-2µs	Low
Key extraction	<1µs	Negligible
Cache lookup	O(1)	Negligible
Pattern matching	O(n) patterns	~1µs

Memory Usage

Per socket: 10-50KB (depends on channels subscribed)
Per channel: max_conflation_keys × max_messages_per_key × avg_message_size

Example:
  100 entities × 100 messages × 1KB = 10MB per channel (high water mark)
  Typical: 10 entities × 20 messages × 500B = 100KB per channel

Encrypted Channel Detection

Delta compression is automatically disabled for private-encrypted-* channels:

Why: Encrypted payloads have no similarity (unique nonces)
Result: Zero compression benefit, wasted CPU cycles
Detection: Automatic - no configuration needed

// This will work, but delta compression is automatically disabled
const channel = pusher.subscribe('private-encrypted-secrets');

// Logs: "Delta compression skipped for encrypted channel"

Best Practices

1. Start Conservative

Begin with global settings, add per-channel optimization later:

{
  "delta_compression": {
    "enabled": true,
    "algorithm": "Fossil"
  }
}

2. Monitor Memory

Adjust cache limits based on actual usage:

{
  "channel_delta_compression": {
    "market-*": {
      "max_messages_per_key": 50,    // Reduce from 100
      "max_conflation_keys": 500      // Reduce from 1000
    }
  }
}

3. Use Patterns

Group similar channels:

{
  "channel_delta_compression": {
    "market-*": "Fossil",
    "iot-*": "Xdelta3",
    "*": "inherit"
  }
}

4. Test First

Verify compression ratio before deploying to production.

Troubleshooting

Delta Compression Not Working

Check 1: Is it enabled in config?

grep -A 5 "delta_compression" config/config.json

Check 2: Did client send enable event? Look for log: “Delta compression enabled for socket” Check 3: Are messages large enough? Default min_message_size is 100 bytes

Cache Sync Not Received

Check 1: Is conflation key configured?

{
  "channel_delta_compression": {
    "market-*": {
      "conflation_key": "asset"  // Must be set
    }
  }
}

Check 2: Client subscribed to channel? Cache sync only sent after successful subscription

Poor Compression Ratio

Symptom: Delta messages as large as original Causes:

Messages differ significantly between updates
Wrong conflation key (not grouping correctly)
Message size too small (< 100 bytes)

Solutions:

Verify conflation key matches your data structure
Check full_message_interval - may be too low
Consider disabling for channels with random data

Migration Guide

From No Compression

1. Add global config:

{
  "delta_compression": {
    "enabled": true
  }
}

2. Update clients:

pusher.connection.bind('connected', () => {
  pusher.connection.send_event('pusher:enable_delta_compression', {});
});

3. Monitor logs for compression statistics

Adding Conflation Keys

1. Identify entity field: e.g., "asset", "device_id" 2. Configure per channel:

{
  "channel_delta_compression": {
    "market-*": {
      "conflation_key": "asset",
      "max_messages_per_key": 100
    }
  }
}

3. Update client to handle cache sync:

channel.bind('pusher:delta_cache_sync', (data) => {
  initializeCache(channel.name, data);
});

4. Test bandwidth improvements (expect 20-40% additional savings)

Next Steps

Tag Filtering

Combine with server-side filtering for maximum efficiency

Rate Limiting

Control message throughput per application

Getting Started

Core Concepts

Configuration

Deployment

Advanced Features

​Overview

​Quick Start

​Server Configuration

​Client Implementation

​Key Features

​Core Capabilities

​Conflation Keys

​Publisher Control

​Client Negotiation

​Compression Algorithms

​Fossil Delta (Default)

​Xdelta3

​Algorithm Selection

​Why Conflation Keys?

​The Problem (Without Conflation)

​The Solution (With Conflation Keys)

​Real-World Impact

​Configuration

​Global Settings

​Per-Channel Configuration

​Simple Format (String-based)

​Full Format (With Conflation Keys)

​Pattern Matching

​Per-Publish Delta Control

​When to Use

​Per-Subscription Delta Negotiation

​Subscription Message Formats

​Priority Order

​Horizontal Scaling

​Node-Local Intervals (Default)

​Cluster Coordination (Optional)

​Use Cases

​✅ Perfect For

​❌ Not Beneficial

​Performance

​Bandwidth Savings

​CPU Overhead

​Memory Usage

​Encrypted Channel Detection

​Best Practices

​1. Start Conservative

​2. Monitor Memory

​3. Use Patterns

​4. Test First

​Troubleshooting

​Delta Compression Not Working

​Cache Sync Not Received

​Poor Compression Ratio

​Migration Guide

​From No Compression

​Adding Conflation Keys

​Next Steps

Tag Filtering

Rate Limiting

Build docs developers (and LLMs) love

Overview

Quick Start

Server Configuration

Client Implementation

Key Features

Core Capabilities

Conflation Keys

Publisher Control

Client Negotiation

Compression Algorithms

Fossil Delta (Default)

Xdelta3

Algorithm Selection

Why Conflation Keys?

The Problem (Without Conflation)

The Solution (With Conflation Keys)

Real-World Impact

Configuration

Global Settings

Per-Channel Configuration

Simple Format (String-based)

Full Format (With Conflation Keys)

Pattern Matching

Per-Publish Delta Control

When to Use

Per-Subscription Delta Negotiation

Subscription Message Formats

Priority Order

Horizontal Scaling

Node-Local Intervals (Default)

Cluster Coordination (Optional)

Use Cases

✅ Perfect For

❌ Not Beneficial

Performance

Bandwidth Savings

CPU Overhead

Memory Usage

Encrypted Channel Detection

Best Practices

1. Start Conservative

2. Monitor Memory

3. Use Patterns

4. Test First

Troubleshooting

Delta Compression Not Working

Cache Sync Not Received

Poor Compression Ratio

Migration Guide

From No Compression

Adding Conflation Keys

Next Steps