Benchmarking

​

Overview

1. Creates a temporary stream in the specified basin
2. Writes records at a target throughput for a specified duration
3. Simultaneously reads records to measure live read performance
4. Waits, then performs a catchup read to measure historical read performance
5. Verifies data integrity using hash chains
6. Reports detailed statistics
7. Deletes the temporary stream

​

Basic usage

s2 bench my-basin

• Record size: 8 KiB
• Target throughput: 1 MiB/s
• Duration: 60 seconds
• Catchup delay: 20 seconds

​

Customize benchmark parameters

​

Record size

s2 bench my-basin --record-size 4096   # 4 KiB records
s2 bench my-basin -b 16384             # 16 KiB records

​

Target throughput

s2 bench my-basin --target-mibps 10    # 10 MiB/s
s2 bench my-basin -t 50                # 50 MiB/s

​

Duration

s2 bench my-basin --duration 120s      # 2 minutes
s2 bench my-basin -d 5m                # 5 minutes

​

Catchup delay

s2 bench my-basin --catchup-delay 30s
s2 bench my-basin -w 1m

​

Storage class

s2 bench my-basin --storage-class express
s2 bench my-basin -c standard

​

Example benchmarks

​

High-throughput test

s2 bench my-basin \
  --record-size 65536 \
  --target-mibps 100 \
  --duration 300s \
  --storage-class express

​

Low-latency test

s2 bench my-basin \
  --record-size 1024 \
  --target-mibps 1 \
  --duration 60s \
  --storage-class express

​

Standard storage test

s2 bench my-basin \
  --record-size 8192 \
  --target-mibps 10 \
  --duration 120s \
  --storage-class standard

​

Understanding benchmark output

Creating temporary stream s2://my-basin/bench/550e8400-e29b-41d4-a716-446655440000 (storage class: express)
Running for 60s targeting 10 MiB/s with 8192 byte records, Ctrl+C to end early

write       10.23 MiB/s     1280 rec/s |   614400000 bytes |       75000 records
read        10.21 MiB/s     1277 rec/s |   613752000 bytes |       74930 records

Write: 10.15 MiB/s, 1269 records/s (614400000 bytes, 75000 records in 57.73s)
Read: 10.12 MiB/s, 1267 records/s (614400000 bytes, 75000 records in 57.89s)

Ack Latency Statistics
min    :      12 ms │ ⠸⠸⠸⠸⠸⠸
median :      18 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸
p90    :      25 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸
p99    :      42 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸
max    :      89 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸

End-to-End Latency Statistics
min    :      15 ms │ ⠸⠸⠸⠸⠸⠸⠸
median :      22 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸
p90    :      31 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸
p99    :      55 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸
max    :     105 ms │ ⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸⠸

Waiting 20s before catchup read...

catchup    102.45 MiB/s    12806 rec/s (614400000 bytes, 75000 records in 5.72s)

​

Metrics explained

​

Throughput

• MiB/s - Megabytes per second (1 MiB = 1,048,576 bytes)
• rec/s - Records per second

​

Latency

• Lower is better for write operations
• Measures write path performance

• Lower is better for real-time applications
• Measures total system latency

​

Statistics

• min - Minimum latency observed
• median - 50th percentile (half of requests faster)
• p90 - 90th percentile (90% of requests faster)
• p99 - 99th percentile (99% of requests faster)
• max - Maximum latency observed

​

Catchup read

• Tests how quickly historical data can be read
• Verifies all written records are readable
• Often shows higher throughput than live reads

​

Data integrity verification

1. Hash chains - Each record’s hash depends on the previous record
2. Record counting - Ensures all written records are read back
3. Body validation - Verifies record body size matches expected

Error: Benchmark verification: unexpected record hash at seq_num 12345

​

Rate limiting

• Calculates expected bytes submitted vs. time elapsed
• Throttles writes to match target MiB/s
• Accounts for network latency and batching

​

Early termination

■ [ABORTED]

1. Stop writing new records
2. Wait for pending acks
3. Verify records written so far
4. Delete the temporary stream

​

Temporary stream naming

bench/{uuid}

• Retention: 1 hour
• Delete on empty: 60 seconds after becoming empty
• Timestamping: Client-required, uncapped

​

Common benchmark scenarios

​

Baseline performance

s2 bench my-basin --duration 300s --storage-class standard > baseline-standard.txt
s2 bench my-basin --duration 300s --storage-class express > baseline-express.txt

​

Compare storage classes

# Standard storage
s2 bench my-basin -c standard -t 10 -d 120s

# Express storage
s2 bench my-basin -c express -t 10 -d 120s

​

Stress test

s2 bench my-basin \
  --record-size 65536 \
  --target-mibps 200 \
  --duration 600s \
  --storage-class express

​

Latency characterization

for THROUGHPUT in 1 5 10 50 100; do
  echo "Testing $THROUGHPUT MiB/s"
  s2 bench my-basin -t $THROUGHPUT -d 60s -c express > bench-${THROUGHPUT}mbps.txt
done

​

Benchmark best practices

​

Run for sufficient duration

• Allow the system to reach steady state
• Collect sufficient samples for accurate statistics
• Average out transient network issues

​

Multiple runs

for i in {1..5}; do
  echo "Run $i"
  s2 bench my-basin -d 120s >> bench-results.txt
  sleep 30
done

​

Consider time of day

• At different times of day
• During peak and off-peak hours
• From different geographic locations

​

Match production workload

# If your app writes 16 KiB records at 5 MiB/s:
s2 bench prod-basin --record-size 16384 --target-mibps 5

​

Troubleshooting

​

Throughput below target

1. Check network bandwidth
2. Enable compression: s2 config set compression zstd
3. Try larger record sizes (reduces per-record overhead)

​

High latency

1. Try express storage class
2. Check network latency to S2 endpoints
3. Run benchmark during off-peak hours

​

Verification errors

1. Re-run the benchmark
2. Check for network issues
3. Contact S2 support if errors persist

​

Out of memory

# Reduce record size
s2 bench my-basin -b 4096 -t 100

​

Interpreting results

​

Good performance indicators

• Write throughput matches target ±5%
• Live read throughput close to write throughput
• Catchup read throughput significantly higher than live read
• p99 latency < 100ms for express storage
• p99 latency < 500ms for standard storage
• Max latency < 2x p99 latency

​

Performance tuning

1. Enable compression (if not already):
   s2 config set compression zstd
   
2. Use express storage for latency-sensitive workloads
3. Optimize record size - Larger records improve throughput efficiency
4. Batch writes in your application (CLI does this automatically)