Custom Database Backends

TAPLE Core provides a trait-based database abstraction that allows you to implement custom storage backends for subjects, events, and other protocol data.

Database Traits

Two traits define the database interface: DatabaseManager for creating collections and DatabaseCollection for data operations.

DatabaseManager

Manages database collections (from core/src/database/mod.rs:11-21):

pub trait DatabaseManager<C>: Sync + Send
where
    C: DatabaseCollection,
{
    /// Default constructor (mainly for testing)
    fn default() -> Self;
    
    /// Creates a database collection
    /// # Arguments
    /// - identifier: The identifier of the collection
    fn create_collection(&self, identifier: &str) -> C;
}

DatabaseCollection

Provides key-value operations with iteration support (from core/src/database/mod.rs:23-37):

pub trait DatabaseCollection: Sync + Send {
    /// Retrieves the value associated with the given key
    fn get(&self, key: &str) -> Result<Vec<u8>, Error>;
    
    /// Associates the given value with the given key
    fn put(&self, key: &str, data: Vec<u8>) -> Result<(), Error>;
    
    /// Removes the value associated with the given key
    fn del(&self, key: &str) -> Result<(), Error>;
    
    /// Returns an iterator over the key-value pairs
    fn iter<'a>(
        &'a self,
        reverse: bool,
        prefix: String,
    ) -> Box<dyn Iterator<Item = (String, Vec<u8>)> + 'a>;
}

Reference Implementation: MemoryManager

TAPLE includes an in-memory implementation for testing and development.

MemoryManager Structure

From core/src/database/memory.rs:40-71:

pub struct MemoryManager {
    data: RwLock<HashMap<String, Arc<DataStore>>>,
}

impl MemoryManager {
    pub fn new() -> Self {
        Self {
            data: RwLock::new(HashMap::new()),
        }
    }
}

impl DatabaseManager<MemoryCollection> for MemoryManager {
    fn default() -> Self {
        Self {
            data: RwLock::new(HashMap::new()),
        }
    }

    fn create_collection(&self, _identifier: &str) -> MemoryCollection {
        let mut lock = self.data.write().unwrap();
        let db: Arc<DataStore> = match lock.get("") {
            Some(map) => map.clone(),
            None => {
                let db: Arc<DataStore> = Arc::new(DataStore::new());
                lock.insert("".to_string(), db.clone());
                db
            }
        };
        MemoryCollection { data: db }
    }
}

MemoryCollection Implementation

From core/src/database/memory.rs:74-110:

pub struct MemoryCollection {
    data: Arc<DataStore>,
}

impl DatabaseCollection for MemoryCollection {
    fn get(&self, key: &str) -> Result<Vec<u8>, Error> {
        let lock = self.data._get_inner_read_lock();
        let Some(data) = lock.get(key) else {
            return Err(Error::EntryNotFound);
        };
        Ok(data.clone())
    }

    fn put(&self, key: &str, data: Vec<u8>) -> Result<(), Error> {
        let mut lock = self.data._get_inner_write_lock();
        lock.insert(key.to_string(), data);
        Ok(())
    }

    fn del(&self, key: &str) -> Result<(), Error> {
        let mut lock = self.data._get_inner_write_lock();
        lock.remove(key);
        Ok(())
    }

    fn iter<'a>(
        &'a self,
        reverse: bool,
        prefix: String,
    ) -> Box<dyn Iterator<Item = (String, Vec<u8>)> + 'a> {
        if reverse {
            Box::new(self.data.rev_iter(prefix))
        } else {
            Box::new(self.data.iter(prefix))
        }
    }
}

DataStore with Iterators

From core/src/database/memory.rs:9-37:

pub struct DataStore {
    data: RwLock<BTreeMap<String, Vec<u8>>>,
}

impl DataStore {
    fn new() -> Self {
        Self {
            data: RwLock::new(BTreeMap::new()),
        }
    }

    fn _get_inner_read_lock<'a>(&'a self) -> RwLockReadGuard<'a, BTreeMap<String, Vec<u8>>> {
        self.data.read().unwrap()
    }

    fn _get_inner_write_lock<'a>(&'a self) -> RwLockWriteGuard<'a, BTreeMap<String, Vec<u8>>> {
        self.data.write().unwrap()
    }

    fn iter(&self, prefix: String) -> MemoryIterator {
        MemoryIterator::new(&self, prefix)
    }

    fn rev_iter(&self, prefix: String) -> RevMemoryIterator {
        RevMemoryIterator::new(&self, prefix)
    }
}

Database Wrapper (DB)

The DB struct provides a high-level interface over database collections for storing TAPLE-specific data.

DB Structure

From core/src/database/db.rs:28-60:

pub struct DB<C: DatabaseCollection> {
    signature_db: SignatureDb<C>,
    subject_db: SubjectDb<C>,
    event_db: EventDb<C>,
    prevalidated_event_db: PrevalidatedEventDb<C>,
    event_request_db: EventRequestDb<C>,
    request_db: RequestDb<C>,
    controller_id_db: ControllerIdDb<C>,
    validation_db: ValidationDb<C>,
    contract_db: ContractDb<C>,
    witness_signatures_db: WitnessSignaturesDb<C>,
    subject_by_governance_db: SubjectByGovernanceDb<C>,
    keys_db: KeysDb<C>,
    preauthorized_subjects_and_providers_db: PreauthorizedSbujectsAndProovidersDb<C>,
    lce_validation_proofs_db: LceValidationProofs<C>,
    approvals_db: ApprovalsDb<C>,
}

Initialization

From core/src/database/db.rs:62-97:

impl<C: DatabaseCollection> DB<C> {
    pub fn new<M: DatabaseManager<C>>(manager: Arc<M>) -> Self {
        let signature_db = SignatureDb::new(&manager);
        let subject_db = SubjectDb::new(&manager);
        let event_db = EventDb::new(&manager);
        let prevalidated_event_db = PrevalidatedEventDb::new(&manager);
        let event_request_db = EventRequestDb::new(&manager);
        let request_db = RequestDb::new(&manager);
        let controller_id_db = ControllerIdDb::new(&manager);
        let validation_db = ValidationDb::new(&manager);
        let contract_db = ContractDb::new(&manager);
        let witness_signatures_db = WitnessSignaturesDb::new(&manager);
        let subject_by_governance_db = SubjectByGovernanceDb::new(&manager);
        let transfer_events_db = KeysDb::new(&manager);
        let preauthorized_subjects_and_providers_db =
            PreauthorizedSbujectsAndProovidersDb::new(&manager);
        let lce_validation_proofs_db = LceValidationProofs::new(&manager);
        let approvals_db = ApprovalsDb::new(&manager);
        
        Self {
            signature_db,
            subject_db,
            event_db,
            prevalidated_event_db,
            event_request_db,
            request_db,
            controller_id_db,
            validation_db,
            contract_db,
            witness_signatures_db,
            subject_by_governance_db,
            keys_db: transfer_events_db,
            preauthorized_subjects_and_providers_db,
            lce_validation_proofs_db,
            approvals_db,
        }
    }
}

Example Operations

From core/src/database/db.rs:129-155:

// Get a subject by ID
pub fn get_subject(&self, subject_id: &DigestIdentifier) -> Result<Subject, Error> {
    self.subject_db.get_subject(subject_id)
}

// Store a subject
pub fn set_subject(
    &self,
    subject_id: &DigestIdentifier,
    subject: Subject,
) -> Result<(), Error> {
    self.subject_db.set_subject(subject_id, subject)
}

// Get subjects with pagination
pub fn get_subjects(
    &self,
    from: Option<String>,
    quantity: isize,
) -> Result<Vec<Subject>, Error> {
    self.subject_db.get_subjects(from, quantity)
}

// Delete a subject
pub fn del_subject(&self, subject_id: &DigestIdentifier) -> Result<(), Error> {
    self.subject_db.del_subject(subject_id)
}

Testing Your Implementation

TAPLE provides a test macro to validate database implementations.

Test Macro

From core/src/database/mod.rs:47-203:

#[macro_export]
macro_rules! test_database_manager_trait {
    ($name:ident: $type:ty: $type2:ty) => {
        mod $name {
            use super::*;
            use borsh::{BorshDeserialize, BorshSerialize};

            #[derive(BorshSerialize, BorshDeserialize, Clone, PartialEq, Eq, Debug)]
            struct Data {
                id: usize,
                value: String,
            }

            #[test]
            fn basic_operations_test() {
                let db = <$type>::default();
                let first_collection: $type2 = db.create_collection("first");
                
                // Test PUT operation
                let result = first_collection.put("a", data[0].clone());
                assert!(result.is_ok());
                
                // Test GET operation
                let result = first_collection.get("a");
                assert!(result.is_ok());
                assert_eq!(result.unwrap(), data[0]);
                
                // Test DEL operation
                let result = first_collection.del("a");
                assert!(result.is_ok());
                
                // Verify deletion
                let result = first_collection.get("a");
                assert!(result.is_err());
            }

            #[test]
            fn iterator_test() {
                let db = <$type>::default();
                let first_collection: $type2 = db.create_collection("first");
                
                // Add test data
                build_state(&first_collection);
                
                // Test forward iteration
                let mut iter = first_collection.iter(false, "".to_string());
                for i in 0..3 {
                    let (key, val) = iter.next().unwrap();
                    assert_eq!(keys[i], key);
                    assert_eq!(data[i], val);
                }
                assert!(iter.next().is_none());
            }

            #[test]
            fn rev_iterator_test() {
                // Test reverse iteration
                let mut iter = first_collection.iter(true, "".to_string());
                for i in (0..3).rev() {
                    let (key, val) = iter.next().unwrap();
                    assert_eq!(keys[i], key);
                    assert_eq!(data[i], val);
                }
                assert!(iter.next().is_none());
            }
        }
    };
}

Use the macro to test your implementation:

test_database_manager_trait! {
    unit_test_my_database:MyDatabaseManager:MyCollection
}

Implementation Guide

Step 1: Define Your Types

pub struct MyDatabaseManager {
    // Your internal state
}

pub struct MyCollection {
    // Your collection state
}

Step 2: Implement DatabaseManager

impl DatabaseManager<MyCollection> for MyDatabaseManager {
    fn default() -> Self {
        // Initialize with defaults
        Self { /* ... */ }
    }

    fn create_collection(&self, identifier: &str) -> MyCollection {
        // Create or get collection
        MyCollection { /* ... */ }
    }
}

Step 3: Implement DatabaseCollection

impl DatabaseCollection for MyCollection {
    fn get(&self, key: &str) -> Result<Vec<u8>, Error> {
        // Retrieve value or return Error::EntryNotFound
    }

    fn put(&self, key: &str, data: Vec<u8>) -> Result<(), Error> {
        // Store value
    }

    fn del(&self, key: &str) -> Result<(), Error> {
        // Delete value
    }

    fn iter<'a>(
        &'a self,
        reverse: bool,
        prefix: String,
    ) -> Box<dyn Iterator<Item = (String, Vec<u8>)> + 'a> {
        // Return iterator over matching keys
    }
}

Step 4: Use with TAPLE

let database = MyDatabaseManager::new();
let (node, api) = Node::build(settings, database)?;

Requirements

Thread Safety

Both traits require Sync + Send
All operations must be thread-safe
Use appropriate locking mechanisms (RwLock, Mutex, etc.)

Iterator Constraints

Iterators must filter by prefix
Support both forward and reverse iteration
Must be valid for lifetime 'a
Return (String, Vec<u8>) tuples

Error Handling

Return Error::EntryNotFound for missing keys
Return Error::SerializeError for serialization failures
Propagate underlying database errors appropriately

The MemoryManager implementation is suitable only for testing and development. For production use, implement a persistent backend like RocksDB, LevelDB, or SQLite.

Performance Considerations

Optimize Hot Paths

get, put, and del are called frequently
Iterator performance affects pagination and queries
Consider caching for frequently accessed data

Batch Operations

While not in the trait, consider implementing:

Batch writes for multiple puts
Transactions for atomic updates
Bulk delete operations

Storage Layout

TAPLE creates these collections:

signatures - Event signatures
subjects - Subject state
events - Event history
contracts - Compiled smart contracts
approvals - Approval requests
And more (see DB struct)

Plan your storage layout accordingly.

Example: LevelDB Backend

use leveldb::database::Database;
use leveldb::kv::KV;
use leveldb::options::{Options, ReadOptions, WriteOptions};

pub struct LevelDbManager {
    path: PathBuf,
}

pub struct LevelDbCollection {
    db: Arc<Database<i32>>,
}

impl DatabaseManager<LevelDbCollection> for LevelDbManager {
    fn default() -> Self {
        Self {
            path: PathBuf::from("./taple_db"),
        }
    }

    fn create_collection(&self, identifier: &str) -> LevelDbCollection {
        let mut options = Options::new();
        options.create_if_missing = true;
        
        let path = self.path.join(identifier);
        let db = Database::open(&path, options).unwrap();
        
        LevelDbCollection {
            db: Arc::new(db),
        }
    }
}

impl DatabaseCollection for LevelDbCollection {
    fn get(&self, key: &str) -> Result<Vec<u8>, Error> {
        let read_opts = ReadOptions::new();
        self.db
            .get(read_opts, key.as_bytes())
            .map_err(|_| Error::DatabaseError)??
            .ok_or(Error::EntryNotFound)
    }

    fn put(&self, key: &str, data: Vec<u8>) -> Result<(), Error> {
        let write_opts = WriteOptions::new();
        self.db
            .put(write_opts, key.as_bytes(), &data)
            .map_err(|_| Error::DatabaseError)
    }

    fn del(&self, key: &str) -> Result<(), Error> {
        let write_opts = WriteOptions::new();
        self.db
            .delete(write_opts, key.as_bytes())
            .map_err(|_| Error::DatabaseError)
    }

    fn iter<'a>(
        &'a self,
        reverse: bool,
        prefix: String,
    ) -> Box<dyn Iterator<Item = (String, Vec<u8>)> + 'a> {
        // Implement LevelDB iterator with prefix filtering
        todo!()
    }
}

Overview

Core Concepts

Getting Started

Guides

Advanced

Custom Database Backends

Database Traits

DatabaseManager

DatabaseCollection

Reference Implementation: MemoryManager

MemoryManager Structure

MemoryCollection Implementation

DataStore with Iterators

Database Wrapper (DB)

DB Structure

Initialization

Example Operations

Testing Your Implementation

Test Macro

Implementation Guide

Step 1: Define Your Types

Step 2: Implement DatabaseManager

Step 3: Implement DatabaseCollection

Step 4: Use with TAPLE

Requirements

Thread Safety

Iterator Constraints

Error Handling

Performance Considerations

Optimize Hot Paths

Batch Operations

Storage Layout

Example: LevelDB Backend

Build docs developers (and LLMs) love

Overview

Core Concepts

Getting Started

Guides

Advanced

​Database Traits

​DatabaseManager

​DatabaseCollection

​Reference Implementation: MemoryManager

​MemoryManager Structure

​MemoryCollection Implementation

​DataStore with Iterators

​Database Wrapper (DB)

​DB Structure

​Initialization

​Example Operations

​Testing Your Implementation

​Test Macro

​Implementation Guide

​Step 1: Define Your Types

​Step 2: Implement DatabaseManager

​Step 3: Implement DatabaseCollection

​Step 4: Use with TAPLE

​Requirements

​Thread Safety

​Iterator Constraints

​Error Handling

​Performance Considerations

​Optimize Hot Paths

​Batch Operations

​Storage Layout

​Example: LevelDB Backend

Build docs developers (and LLMs) love

Database Traits

DatabaseManager

DatabaseCollection

Reference Implementation: MemoryManager

MemoryManager Structure

MemoryCollection Implementation

DataStore with Iterators

Database Wrapper (DB)

DB Structure

Initialization

Example Operations

Testing Your Implementation

Test Macro

Implementation Guide

Step 1: Define Your Types

Step 2: Implement DatabaseManager

Step 3: Implement DatabaseCollection

Step 4: Use with TAPLE

Requirements

Thread Safety

Iterator Constraints

Error Handling

Performance Considerations

Optimize Hot Paths

Batch Operations

Storage Layout

Example: LevelDB Backend