Skip to main content
TieringConfig allows you to tier cold tree levels to memory-mapped data files after checkpoints, reducing memory usage for large trees while maintaining hot data in memory for fast access.

​
Definition

Defined in src/storage/checkpoint.rs:58 
pub struct TieringConfig {
    /// Levels below this value have their committed chunks mmap'd after checkpoint.
    /// Set to `usize::MAX` to mmap all levels (default), `0` to keep everything in memory
    pub pin_above_level: usize,
}

​
Fields

​
pin_above_level
usize
required
Controls the memory/disk boundary:
  • Levels ≥ pin_above_level: Kept in memory (hot data)
  • Levels < pin_above_level: mmap’d from disk files after checkpoint (cold data)
Special values:
  • 0 - Keep everything in memory (no mmap)
  • usize::MAX - mmap all levels (default, minimum memory)
Typical values:
  • 0-2 - Keep top 2 levels in memory (recent inserts hot)
  • 3-5 - Moderate tiering
  • usize::MAX - Maximum memory savings

​
How Tiering Works

​
Tree Level Structure

In an n-ary tree with branching factor N:
  • Level 0: Leaf level (largest, coldest)
  • Level 1: First parent level
  • Level 2+: Higher parent levels
  • Top level: Root (smallest, hottest)
Higher level numbers = closer to root = smaller size = hotter access patterns.Level 0 (leaves) is the largest and least frequently accessed.

​
Memory vs Mmap

After a checkpoint: 
pin_above_level = 2

    Level 4 (root)      ← In memory (4 ≥ 2)
    Level 3             ← In memory (3 ≥ 2)
    Level 2             ← In memory (2 ≥ 2)
    Level 1             ← mmap'd (1 < 2)
    Level 0 (leaves)    ← mmap'd (0 < 2)
Writes always go to memory. On checkpoint, levels below pin_above_level are remapped to mmap’d files.

​
Usage Examples

​
Default: Maximize Memory Savings

use rotortree::TieringConfig;

let config = RotorTreeConfig {
    // ...
    tiering: TieringConfig::default(), // pin_above_level = usize::MAX
    // ...
};

// All checkpointed levels are mmap'd
// Minimum memory usage

​
Keep Everything in Memory

use rotortree::TieringConfig;

let config = RotorTreeConfig {
    // ...
    tiering: TieringConfig { pin_above_level: 0 },
    // ...
};

// Nothing is mmap'd
// Maximum performance, maximum memory usage

​
Hybrid: Hot Levels in Memory

use rotortree::TieringConfig;

let config = RotorTreeConfig {
    // ...
    tiering: TieringConfig { pin_above_level: 2 },
    // ...
};

// Levels 0 and 1 are mmap'd (cold)
// Levels ≥ 2 stay in memory (hot)
// Good balance for large trees

​
Memory Usage Examples

Each chunk is 128 hashes × 32 bytes = 4 KiB.Level sizes decrease by factor of N at each level.

​
N=4, MAX_DEPTH=16, 1B Leaves

Approximate level sizes: 
Level  0: ~32 GiB  (leaves)
Level  1: ~8 GiB
Level  2: ~2 GiB
Level  3: ~512 MiB
Level  4: ~128 MiB
Level  5: ~32 MiB
Level 6+: <32 MiB total
Tiering options:
pin_above_levelMemory Usagemmap’d SizeProfile
0~43 GiB0All memory
1~11 GiB~32 GiBHot path in memory
2~3 GiB~40 GiBBalanced
3~700 MiB~42 GiBMinimal memory
usize::MAX<100 MiB~43 GiBMaximum savings

​
N=8, MAX_DEPTH=10, 1B Leaves

Approximate level sizes: 
Level 0: ~32 GiB  (leaves)
Level 1: ~4 GiB
Level 2: ~512 MiB
Level 3: ~64 MiB
Level 4+: <64 MiB total
Higher branching factors concentrate data in lower levels.

​
Performance Characteristics

​
Read Performance

Data LocationAccess LatencyThroughput
In-memory~10-50nsMaximum
mmap (hot page cache)~100-500nsHigh
mmap (cold, SSD)~10-100µsMedium
mmap (cold, HDD)~5-10msLow
The OS page cache keeps frequently accessed mmap’d pages in RAM. After warmup, mmap performance approaches in-memory performance for hot data.

​
Write Performance

Writes always go to in-memory WAL buffer first, so tiering doesn’t affect write latency. Impact is only on checkpoint operations.

​
Choosing pin_above_level

Small Trees (<100M leaves)

Recommendation: pin_above_level = 0Keep everything in memory. Memory usage is reasonable and performance is maximized.

Medium Trees (100M-1B leaves)

Recommendation: pin_above_level = 2Balance memory and performance. Top levels hot, bottom levels cold.

Large Trees (>1B leaves)

Recommendation: pin_above_level = 3 or usize::MAXMinimize memory usage. Rely on OS page cache for frequently accessed data.

Proof Generation Workload

Recommendation: pin_above_level = 1 or 2Proof generation reads from all levels. Keeping upper levels hot improves latency.

​
Monitoring Memory Usage

Monitor actual memory usage with OS tools: 
# Linux: Check process RSS
ps aux | grep rotortree

# macOS: Check memory
top -pid <PID>

# Detailed memory map
pmap <PID>

​
Caveats

  • mmap is not available on all platforms: Windows support varies
  • File handle limits: Each shard file requires a file handle
  • Page cache pressure: Large mmap’d regions can evict other cached data
  • Checkpoint overhead: Remapping levels adds latency to checkpoint operations

​
Default Value

impl Default for TieringConfig {
    fn default() -> Self {
        Self {
            pin_above_level: usize::MAX, // mmap everything
        }
    }
}

​
Shard Files

After checkpointing, data files are organized as: 
data/
├── header.bin
├── checkpoint.meta
├── tails.bin
├── level_0/
│   ├── shard_0000.dat
│   ├── shard_0001.dat
│   └── ...
├── level_1/
│   ├── shard_0000.dat
│   └── ...
└── level_2/
    └── ...
Each shard file contains up to 65,536 chunks (256 MiB).

Build docs developers (and LLMs) love

Get started for freeTalk to us