Durability

​

Durability guarantee

• No acknowledged data loss: If you receive an ack, the data is durable
• No silent failures: Failed writes return errors, not acknowledgments
• Crash recovery: Server crashes cannot lose acknowledged data
• Consistent reads: Acknowledged writes are immediately visible to all readers

​

Write path durability

​

Write implementation

async fn db_write_records(
    db: slatedb::Db,
    stream_id: StreamId,
    retention: RetentionPolicy,
    records: Vec<Metered<SequencedRecord>>,
    // ...
) -> Result<Vec<Metered<SequencedRecord>>, slatedb::Error> {
    // Build write batch with all records
    let mut wb = WriteBatch::new();
    for (position, record) in records.iter().map(|msr| msr.parts()) {
        wb.put_with_options(
            kv::stream_record_data::ser_key(stream_id, position),
            kv::stream_record_data::ser_value(record),
            &ttl_put_opts,
        );
        // Also write timestamp index
        wb.put_with_options(
            kv::stream_record_timestamp::ser_key(stream_id, position),
            kv::stream_record_timestamp::ser_value(),
            &ttl_put_opts,
        );
    }
    // Update tail position
    wb.put(
        kv::stream_tail_position::ser_key(stream_id),
        kv::stream_tail_position::ser_value(next_pos(&records), write_timestamp_secs),
    );
    
    // Write with await_durable flag
    static WRITE_OPTS: WriteOptions = WriteOptions {
        await_durable: true,
    };
    db.write_with_options(wb, &WRITE_OPTS).await?;
    Ok(records)
}

1. Atomic batch writes: All records in a batch are written together
2. await_durable flag: Write only returns after durability is confirmed
3. Tail position update: Included in the same atomic write

​

SlateDB durability

• LSM-tree structure: Write-optimized log-structured merge tree
• Object storage backend: All data stored in S3-compatible object storage
• Write-ahead log: Changes are logged before being applied
• Flush intervals: Configurable intervals for flushing to object storage

It uses SlateDB as its storage engine, which relies entirely on object storage for durability. Just like s2.dev, data is always durable on object storage before being acknowledged or returned to readers.

​

Configuring flush intervals

# Default: 50ms for remote bucket, 5ms for in-memory
SL8_FLUSH_INTERVAL=10ms s2 lite --bucket my-bucket

​

Storage layout

​

Record data

/// (SRD) per-record, immutable
/// Key: StreamID StreamPosition
/// Value: EnvelopeRecord
StreamRecordData(StreamId, StreamPosition),

• Key: Stream ID + sequence number + timestamp
• Value: The encoded record (headers + body)
• Immutable: Never updated after write
• TTL: Respects retention policy

​

Record timestamp index

/// (SRT) per-record, immutable
/// Key: StreamID Timestamp SeqNum
/// Value: empty
StreamRecordTimestamp(StreamId, StreamPosition),

• Key: Stream ID + timestamp + sequence number
• Value: Empty (existence is sufficient)
• Purpose: Enables timestamp-based reads
• TTL: Same as record data

​

Tail position

/// (SP) per-stream, updatable
/// Key: StreamID
/// Value: SeqNum Timestamp WriteTimestampSecs
StreamTailPosition(StreamId),

• Atomically updated with each append
• Includes write timestamp for freshness tracking
• Used to determine the next sequence number

​

Retention and TTL

let ttl = match retention {
    RetentionPolicy::Age(age) => Ttl::ExpireAfter(age.as_millis() as u64),
    RetentionPolicy::Infinite() => Ttl::NoExpiry,
};

• Age-based: Records expire after the configured duration
• Infinite: Records never expire automatically
• TTL enforcement: SlateDB handles expiration during compaction

​

Pipelining and performance

append_futs: FuturesOrdered<BoxFuture<'static, Result<Vec<...>, slatedb::Error>>>,
pending_appends: append::PendingAppends::new(),
stable_pos: tail_pos,

• Multiple in-flight appends: Appends pipeline to object storage
• Ordered completion: Results are processed in order via FuturesOrdered
• Backpressure: Semaphore limits total in-flight bytes

Pipelining is temporarily disabled by default, and it will be enabled once it is completely safe. For now, you can use S2LITE_PIPELINE=true to get a sense of what performance will look like.

​

Backpressure mechanism

let append_inflight_bytes_max =
    (append_inflight_max.as_u64() as usize).min(Semaphore::MAX_PERMITS);

let append_inflight_bytes_sema = Arc::new(Semaphore::new(append_inflight_bytes_max));

let metered_size = input.records.metered_size();
let num_permits = metered_size.clamp(1, self.append_inflight_bytes_max) as u32;
let sema_permit = tokio::select! {
    res = self.append_inflight_bytes_sema.acquire_many(num_permits) => {
        res.map_err(|_| StreamerMissingInActionError)
    }
    _ = self.msg_tx.closed() => {
        Err(StreamerMissingInActionError)
    }
}?;

​

Failure scenarios

​

Client fails before receiving ack

• Records are durable and assigned sequence numbers
• Client doesn’t know if append succeeded
• Client can use match_seq_num on retry to detect duplicate

​

Server crashes after write

• Records are durable on object storage
• On restart, server loads tail position from storage
• All durable records are visible to readers

​

Object storage write fails

• Error propagated to streamer
• All pending appends for that write fail
• Client receives error, knows append did not succeed
• Client can safely retry

Err(db_err) => {
    self.pending_appends.on_durability_failed(db_err);
    break;
}

​

Network partition

• Writes fail and return errors
• No acknowledgments sent to clients
• No data loss occurs
• Service resumes when partition heals

​

Comparing with other systems

​

Monitoring durability

​

Write latency

curl http://localhost:8080/metrics | grep append_ack_latency

• Measures time from append request to ack
• Includes durability confirmation
• Should correlate with object storage latency

​

Flush metrics

• flush_interval: Configured flush interval
• flush_duration: Actual time to complete flush
• flush_bytes: Bytes flushed per interval

​

Health checks

curl http://localhost:8080/health

• Server is running
• Can accept requests
• SlateDB is operational

​

Best practices

1. Trust the ack: Only consider writes successful when ack is received
2. Handle errors: Implement retry logic for failed appends
3. Use match_seq_num: Prevent duplicates on retry with optimistic concurrency
4. Monitor latency: Watch write latency to detect object storage issues
5. Configure flush interval: Tune based on latency/throughput requirements
6. Test failure scenarios: Verify application handles partial failures correctly

​

Object storage options

​

AWS S3

docker run -p 8080:80 \
  -e AWS_PROFILE=${AWS_PROFILE} \
  -v ~/.aws:/home/nonroot/.aws:ro \
  ghcr.io/s2-streamstore/s2 lite \
  --bucket ${S3_BUCKET} \
  --path s2lite

​

Tigris, Cloudflare R2, etc.

docker run -p 8080:80 \
  -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
  -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  -e AWS_ENDPOINT_URL_S3=${AWS_ENDPOINT_URL_S3} \
  ghcr.io/s2-streamstore/s2 lite \
  --bucket ${S3_BUCKET} \
  --path s2lite

​

Local disk

s2 lite --local-root ./data

​

In-memory (testing)

s2 lite --port 8080

​

Next steps