SstFileWriter - RocksDB

SstFileWriter creates SST (Sorted String Table) files that can be efficiently ingested into a RocksDB database. This is useful for bulk loading data, as it avoids the overhead of individual Put operations.

Creating an SstFileWriter

Constructor

SstFileWriter(
  const EnvOptions& env_options,
  const Options& options,
  ColumnFamilyHandle* column_family = nullptr,
  bool invalidate_page_cache = true,
  Env::IOPriority io_priority = Env::IOPriority::IO_TOTAL
);

env_options

const EnvOptions&

Environment options for file operations.

options

const Options&

Database options including comparator and table options.

column_family

ColumnFamilyHandle*

Target column family. Pass nullptr if unknown.

invalidate_page_cache

bool

If true, hints to OS that file pages are not needed after writing.

io_priority

Env::IOPriority

I/O priority for rate limiting.

Opening a File

Open

Status Open(
  const std::string& file_path,
  Temperature temp = Temperature::kUnknown
);

Prepares SstFileWriter to write into the specified file.

file_path

const std::string&

Path where the SST file will be created.

temp

Temperature

Temperature hint for storage placement (kHot, kWarm, kCold).

Status

Returns OK on success.

Adding Data

Put

Status Put(const Slice& user_key, const Slice& value);

Adds a key-value pair to the SST file.

user_key

const Slice&

Key to add. Must be after any previously added key according to the comparator.

value

const Slice&

Value to associate with the key.

Status

Returns OK on success, InvalidArgument if key order is violated.

Put with Timestamp

Status Put(
  const Slice& user_key,
  const Slice& timestamp,
  const Slice& value
);

Adds a key with user-defined timestamp.

timestamp

const Slice&

User-defined timestamp. Size must match comparator expectations.

PutEntity

Status PutEntity(
  const Slice& user_key,
  const WideColumns& columns
);

Adds a wide-column entity.

columns

const WideColumns&

Wide-column entity data.

Merge

Status Merge(const Slice& user_key, const Slice& value);

Adds a merge operand.

Delete

Status Delete(const Slice& user_key);

Adds a deletion marker.

Delete with Timestamp

Status Delete(const Slice& user_key, const Slice& timestamp);

Adds a deletion marker with timestamp.

DeleteRange

Status DeleteRange(const Slice& begin_key, const Slice& end_key);

Adds a range deletion tombstone [begin_key, end_key).

begin_key

const Slice&

Start of the range (inclusive).

end_key

const Slice&

End of the range (exclusive).

Finalizing the File

Finish

Status Finish(ExternalSstFileInfo* file_info = nullptr);

Finalizes writing and closes the SST file.

file_info

ExternalSstFileInfo*

Optional output parameter populated with information about the created file.

Status

Returns OK on success.

FileSize

uint64_t FileSize();

Returns the current file size.

uint64_t

Current size of the SST file in bytes.

File Information

ExternalSstFileInfo

struct ExternalSstFileInfo {
  std::string file_path;
  std::string smallest_key;
  std::string largest_key;
  std::string smallest_range_del_key;
  std::string largest_range_del_key;
  std::string file_checksum;
  std::string file_checksum_func_name;
  SequenceNumber sequence_number;
  uint64_t file_size;
  uint64_t num_entries;
  uint64_t num_range_del_entries;
  int32_t version;
};

file_path

std::string

Path to the created SST file.

smallest_key

std::string

Smallest user key in the file.

largest_key

std::string

Largest user key in the file.

sequence_number

SequenceNumber

Sequence number of all keys in the file (always 0 for SstFileWriter).

file_size

uint64_t

File size in bytes.

num_entries

uint64_t

Number of entries in the file.

Example: Basic Usage

#include "rocksdb/sst_file_writer.h"
#include "rocksdb/db.h"

using namespace ROCKSDB_NAMESPACE;

// Create SST file writer
Options options;
options.create_if_missing = true;
EnvOptions env_options;

SstFileWriter sst_writer(env_options, options);

// Open file for writing
std::string file_path = "/tmp/test.sst";
Status s = sst_writer.Open(file_path);
assert(s.ok());

// Add sorted key-value pairs
s = sst_writer.Put("key1", "value1");
assert(s.ok());

s = sst_writer.Put("key2", "value2");
assert(s.ok());

s = sst_writer.Put("key3", "value3");
assert(s.ok());

// Finish writing
ExternalSstFileInfo file_info;
s = sst_writer.Finish(&file_info);
assert(s.ok());

printf("Created SST file: %s\n", file_info.file_path.c_str());
printf("File size: %lu bytes\n", file_info.file_size);
printf("Number of entries: %lu\n", file_info.num_entries);
printf("Smallest key: %s\n", file_info.smallest_key.c_str());
printf("Largest key: %s\n", file_info.largest_key.c_str());

Example: Bulk Loading with Ingestion

#include "rocksdb/sst_file_writer.h"
#include "rocksdb/db.h"

using namespace ROCKSDB_NAMESPACE;

// Open database
DB* db;
Options options;
options.create_if_missing = true;
Status s = DB::Open(options, "/tmp/testdb", &db);
assert(s.ok());

// Create SST file
SstFileWriter sst_writer(EnvOptions(), options);
s = sst_writer.Open("/tmp/bulk_load.sst");
assert(s.ok());

// Add 1 million sorted entries
for (int i = 0; i < 1000000; i++) {
  char key[32], value[100];
  snprintf(key, sizeof(key), "key%07d", i);
  snprintf(value, sizeof(value), "value%07d", i);
  s = sst_writer.Put(key, value);
  assert(s.ok());
}

ExternalSstFileInfo file_info;
s = sst_writer.Finish(&file_info);
assert(s.ok());

// Ingest the SST file
IngestExternalFileOptions ingest_options;
s = db->IngestExternalFile({file_info.file_path}, ingest_options);
assert(s.ok());

printf("Successfully ingested %lu entries\n", file_info.num_entries);

delete db;

Example: Range Deletions

#include "rocksdb/sst_file_writer.h"

using namespace ROCKSDB_NAMESPACE;

Options options;
SstFileWriter sst_writer(EnvOptions(), options);
Status s = sst_writer.Open("/tmp/range_del.sst");

// Add some entries
sst_writer.Put("a", "value_a");
sst_writer.Put("b", "value_b");
sst_writer.Put("e", "value_e");
sst_writer.Put("f", "value_f");

// Add range deletion [b, e)
// This will delete keys b, c, d (but not e)
sst_writer.DeleteRange("b", "e");

ExternalSstFileInfo file_info;
s = sst_writer.Finish(&file_info);
assert(s.ok());

printf("Range deletions: %lu\n", file_info.num_range_del_entries);

Key Ordering Requirements

Sorted Keys Required

Keys must be added in sorted order according to the comparator:

SstFileWriter writer(EnvOptions(), options);
writer.Open("/tmp/test.sst");

// Correct order
writer.Put("apple", "1");   // OK
writer.Put("banana", "2");  // OK
writer.Put("cherry", "3");  // OK

// Wrong order - will fail
writer.Put("aardvark", "0"); // ERROR: out of order

Range Deletions Are Order-Independent

Range deletion tombstones can be added in any order:

writer.DeleteRange("m", "p");  // OK
writer.DeleteRange("a", "d");  // OK - can be out of order
writer.DeleteRange("x", "z");  // OK

Verification

CreatedBySstFileWriter

static bool CreatedBySstFileWriter(
  const TableProperties& table_props
);

Checks if a file with the given table properties was created by SstFileWriter.

table_props

const TableProperties&

Table properties to check.

bool

Returns true if the file was created by SstFileWriter.

Performance Tips

Buffer Size

Larger write buffers can improve performance:

Options options;
options.write_buffer_size = 64 * 1024 * 1024;  // 64MB
SstFileWriter writer(EnvOptions(), options);

Compression

Enable compression to reduce file size:

Options options;
options.compression = CompressionType::kLZ4Compression;
SstFileWriter writer(EnvOptions(), options);

Page Cache Invalidation

Enable page cache invalidation for better memory usage:

SstFileWriter writer(EnvOptions(), options,
                     nullptr,   // column_family
                     true);     // invalidate_page_cache

Common Pitfalls

Unsorted Keys

The most common error is adding keys out of order:

// WRONG - keys not sorted
writer.Put("zebra", "1");
writer.Put("apple", "2");  // ERROR

Always ensure keys are sorted before adding.

Forgetting to Call Finish

Always call Finish() before using the file:

writer.Open("/tmp/test.sst");
writer.Put("key", "value");
// WRONG - file is incomplete

// CORRECT
writer.Finish();

Sequence Numbers

All keys in SstFileWriter files have sequence number 0. When ingested, they receive the current database sequence number.

Database

Read & Write

Transactions

Storage

Utilities

​Creating an SstFileWriter

​Constructor

​Opening a File

​Open

​Adding Data

​Put

​Put with Timestamp

​PutEntity

​Merge

​Delete

​Delete with Timestamp

​DeleteRange

​Finalizing the File

​Finish

​FileSize

​File Information

​ExternalSstFileInfo

​Example: Basic Usage

​Example: Bulk Loading with Ingestion

​Example: Range Deletions

​Key Ordering Requirements

​Sorted Keys Required

​Range Deletions Are Order-Independent

​Verification

​CreatedBySstFileWriter

​Performance Tips

​Buffer Size

​Compression

​Page Cache Invalidation

​Common Pitfalls

​Unsorted Keys

​Forgetting to Call Finish

​Sequence Numbers

​See Also

Build docs developers (and LLMs) love

Creating an SstFileWriter

Constructor

Opening a File

Open

Adding Data

Put

Put with Timestamp

PutEntity

Merge

Delete

Delete with Timestamp

DeleteRange

Finalizing the File

Finish

FileSize

File Information

ExternalSstFileInfo

Example: Basic Usage

Example: Bulk Loading with Ingestion

Example: Range Deletions

Key Ordering Requirements

Sorted Keys Required

Range Deletions Are Order-Independent

Verification

CreatedBySstFileWriter

Performance Tips

Buffer Size

Compression

Page Cache Invalidation

Common Pitfalls

Unsorted Keys

Forgetting to Call Finish

Sequence Numbers

See Also