Skip to main content
Ubicloud provides virtualized block storage for VMs using the Storage Performance Development Kit (SPDK). SPDK enables enterprise features such as encryption, snapshots, and low-latency I/O operations.

​
Architecture

The block storage system uses SPDK to deliver high-performance storage to virtual machines:
  • SPDK Version: v23.09-ubi-0.3 with custom bdev_ubi block device support
  • Userspace Driver: SPDK runs in userspace for maximum performance
  • Hugepages: 1GB of hugepages per CPU core for zero-copy operations
  • Vhost Interface: VMs connect to storage via vhost-blk controllers

​
SPDK Installation

Block storage installations are managed per VM host. Each installation tracks: 
# From model/spdk_installation.rb
class SpdkInstallation < Sequel::Model
  many_to_one :vm_host
  one_to_many :vm_storage_volumes
  
  # Columns:
  #  version           - SPDK version (e.g., "v23.09-ubi-0.3")
  #  cpu_count         - Number of CPU cores (default: 2)
  #  hugepages         - Hugepages in GB (default: 2)
  #  allocation_weight - For load balancing across hosts
end
Key Implementation Details (spdk_installation.rb:11-15): 
def supports_bdev_ubi?
  # Custom bdev_ubi support for advanced features
  version.match?(/^v[0-9]+\.[0-9]+-ubi-.*/)  
end

​
Configuration

​
I/O Buffer Configuration

SPDK pre-allocates memory pools for efficient I/O operations. The configuration scales with CPU count: 
# From rhizome/host/lib/spdk_setup.rb:140-159
small_pool_count = 19200 * cpu_count  # 8KB buffers
large_pool_count = 2400 * cpu_count   # 135KB buffers

iobuf_conf = [{
  method: "iobuf_set_options",
  params: {
    # Supports 18 VMs on AX-162 doing bursts of 256k writes
    small_pool_count: small_pool_count,
    large_pool_count: large_pool_count,
    small_bufsize: 8192,
    large_bufsize: 135168
  }
}]

​
Block Device Configuration

The bdev subsystem manages block devices and I/O caching: 
bdev_conf = [{
  method: "bdev_set_options",
  params: {
    bdev_io_pool_size: 65536,      # Supports 512 volumes per host
    bdev_io_cache_size: 256,       # Cache per io_channel
    bdev_auto_examine: true
  }
}]

​
Service Management

​
Systemd Service

SPDK runs as a systemd service with the following configuration (spdk_setup.rb:81-113): 
[Unit]
Description=Block Storage Service v23.09-ubi-0.3
Requires=spdk-hugepages.mount

[Service]
Type=simple
Environment="XDG_RUNTIME_DIR=/home/spdk"
ExecStart=/opt/ubicloud/spdk/v23.09-ubi-0.3/bin/vhost_ubi \
  -S /var/lib/spdk/vhost \
  --huge-dir /home/spdk/hugepages \
  --iova-mode va \
  --rpc-socket /var/run/spdk.sock \
  --cpumask [0,1] \
  --disable-cpumask-locks \
  --config /etc/spdk/config.json

LimitMEMLOCK=8400113664
PrivateDevices=yes
PrivateTmp=yes
ProtectKernelTunables=yes
User=spdk
Group=spdk

​
Hugepages Mount

Hugepages are mounted with 1GB per CPU core: 
[Mount]
What=hugetlbfs
Where=/home/spdk/hugepages/v23.09-ubi-0.3
Type=hugetlbfs
Options=uid=spdk,size=2G

​
Volume Management

​
Creating Storage Volumes

Storage volumes are attached to VMs with encryption and QoS support: 
# From model/vm_storage_volume.rb
class VmStorageVolume < Sequel::Model
  many_to_one :vm
  many_to_one :spdk_installation
  many_to_one :key_encryption_key_1  # Primary KEK
  many_to_one :key_encryption_key_2  # Secondary KEK
  
  # Columns:
  #  size_gib                 - Volume size in GB
  #  disk_index               - Device index (0 for boot)
  #  boot                     - Is this a boot volume?
  #  use_bdev_ubi             - Use custom bdev_ubi driver
  #  max_read_mbytes_per_sec  - Read bandwidth limit
  #  max_write_mbytes_per_sec - Write bandwidth limit
end

​
SPDK RPC Operations

Volumes are created using SPDK RPC calls:
1

Create AIO Block Device

Create the underlying block device from a file:
# From rhizome/host/lib/spdk_rpc.rb:12-20
def bdev_aio_create(name, filename, block_size)
  params = {
    name: name,
    filename: filename,
    block_size: block_size,
    readonly: false
  }
  call("bdev_aio_create", params)
end
2

Add Encryption Layer

Wrap the block device with encryption:
# From spdk_rpc.rb:28-35
def bdev_crypto_create(name, base_bdev_name, key_name)
  params = {
    name: name,
    base_bdev_name: base_bdev_name,
    key_name: key_name
  }
  call("bdev_crypto_create", params)
end
3

Create Vhost Controller

Expose the volume to the VM:
# From spdk_rpc.rb:65-71
def vhost_create_blk_controller(name, bdev)
  params = {
    ctrlr: name,
    dev_name: bdev
  }
  call("vhost_create_blk_controller", params)
end
4

Set QoS Limits

Configure bandwidth limits:
# From spdk_rpc.rb:95-100
def bdev_set_qos_limit(name, rw_ios_per_sec: nil, 
                       r_mbytes_per_sec: nil, 
                       w_mbytes_per_sec: nil)
  params = {
    name: name,
    rw_ios_per_sec: rw_ios_per_sec || 0,
    r_mbytes_per_sec: r_mbytes_per_sec || 0,
    w_mbytes_per_sec: w_mbytes_per_sec || 0
  }
  call("bdev_set_qos_limit", params)
end

​
Custom bdev_ubi Driver

The bdev_ubi driver provides advanced copy-on-write and caching capabilities: 
# From spdk_rpc.rb:43-57
def bdev_ubi_create(name, base_bdev_name, image_path,
  stripe_size_kb = 1024,
  copy_on_read = false,
  directio = true)
  params = {
    name: name,
    base_bdev: base_bdev_name,
    image_path: image_path,
    stripe_size_kb: stripe_size_kb,
    no_sync: false,
    copy_on_read: copy_on_read,
    directio: directio
  }
  call("bdev_ubi_create", params)
end
Parameters:
  • stripe_size_kb: Chunk size for striping (default: 1024KB)
  • copy_on_read: Enable COW on read operations
  • directio: Bypass page cache for direct I/O

​
Performance Characteristics

​
Memory Allocation

​
small_pool_count
integer
Number of 8KB I/O buffers. Set to 19200 * cpu_count to support concurrent I/O.SPDK pre-allocates 256 small items per encrypted volume.
​
large_pool_count
integer
Number of 135KB I/O buffers. Set to 2400 * cpu_count.Peaks at 512 large items per volume during 256KB write bursts.

​
Capacity Planning

  • VMs per Host: 18 concurrent VMs on AX-162 hardware
  • Volumes per Host: Up to 512 volumes with default configuration
  • I/O Channels: One channel per CPU core assigned to SPDK
  • Hugepages: 1GB per CPU core

​
Installation Process

1

Install Dependencies

apt-get -y install libaio-dev libssl-dev libnuma-dev \
  libjson-c-dev uuid-dev libiscsi-dev
2

Create SPDK User

adduser spdk --disabled-password --gecos '' --home /home/spdk
mkdir -p /var/lib/spdk/vhost
chown spdk:spdk /var/lib/spdk/vhost
3

Download and Extract Package

curl -L3 -o /tmp/spdk.tar.gz \
  https://github.com/ubicloud/bdev_ubi/releases/download/spdk-23.09-ubi-0.3/ubicloud-spdk-ubuntu24.04-x64.tar.gz

mkdir -p /opt/ubicloud/spdk/v23.09-ubi-0.3
tar -xzf /tmp/spdk.tar.gz --strip-components=1 \
  -C /opt/ubicloud/spdk/v23.09-ubi-0.3
4

Configure and Enable Services

# Create configuration in /etc/spdk/config.json
# Create systemd units
systemctl enable spdk-hugepages.mount
systemctl enable spdk.service

systemctl start spdk-hugepages.mount
systemctl start spdk.service
5

Verify Installation

systemctl is-active spdk.service
# Output: active
SPDK requires dedicated CPU cores and hugepages. Ensure your host has sufficient resources before installation.

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us