Filesystem VFS

Overview

Oboromi implements a minimal virtual filesystem (VFS) layer to abstract file I/O operations. The current implementation focuses on memory-mapped file access for performance-critical ROM loading and asset streaming.

Architecture

The filesystem layer is designed to support multiple storage backends while providing a unified API:

┌──────────────────┐
│   Game Code      │
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│   VFS Layer      │  ◄── Future: Multiple backends
└────────┬─────────┘
         │
         ▼
┌──────────────────┐
│  Memory-Mapped   │  ◄── Current implementation
│      Files       │
└──────────────────┘

File Interface

The core filesystem type is a memory-mapped file wrapper (defined in core/src/fs/mod.rs):

use memmap2::Mmap;
use std::fs;
use std::path::Path;
use std::ops::Deref;

pub struct File {
    map: Mmap,
}

Opening Files

impl File {
    pub fn open<P: AsRef<Path>>(path: P) -> Result<Self, std::io::Error>
    where
    {
        let file = fs::File::open(path)?;      
        let map = unsafe { Mmap::map(&file)? };
        Ok(Self { map })
    }
}

Key characteristics:

Takes any type implementing AsRef<Path> (strings, Path, PathBuf)
Uses memmap2 crate for efficient memory mapping
Maps entire file into process address space
Returns std::io::Error on failure

The Mmap::map() call is marked unsafe because memory-mapped files can cause undefined behavior if the underlying file is modified or deleted while mapped.

Deref Implementation

impl Deref for File {
    type Target = [u8];

    fn deref(&self) -> &Self::Target {
        &self.map
    }
}

The Deref trait allows File to be used like a byte slice:

let file = File::open("game.rom")?;

// Direct indexing
let first_byte = file[0];

// Slicing
let header = &file[0..16];

// Length
let size = file.len();

// Iteration
for byte in file.iter() {
    // process byte
}

Usage Examples

Loading a ROM File

use oboromi_core::fs::File;

fn load_rom(path: &str) -> Result<(), std::io::Error> {
    let rom = File::open(path)?;
    
    // Check magic number
    if &rom[0..4] != b"NSP\0" {
        eprintln!("Invalid ROM format");
        return Err(std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            "Not a valid NSP file"
        ));
    }
    
    // Read ROM size from header
    let size = u32::from_le_bytes([
        rom[4], rom[5], rom[6], rom[7]
    ]);
    
    println!("ROM size: {} bytes", size);
    Ok(())
}

Parsing File Structures

use std::mem;

#[repr(C, packed)]
struct NcaHeader {
    magic: [u8; 4],
    fixed_key_sig: [u8; 256],
    npdm_sig: [u8; 256],
    // ... more fields ...
}

fn parse_nca_header(file: &File) -> Result<&NcaHeader, &'static str> {
    if file.len() < mem::size_of::<NcaHeader>() {
        return Err("File too small for NCA header");
    }
    
    // SAFETY: We verified the size above
    unsafe {
        let ptr = file.as_ptr() as *const NcaHeader;
        Ok(&*ptr)
    }
}

When casting memory-mapped data to structured types, ensure proper alignment and size validation to avoid undefined behavior.

Memory Mapping Benefits

Zero-Copy

Data is accessed directly from the kernel page cache without copying to userspace buffers.

Lazy Loading

Pages are loaded on-demand as accessed, reducing startup time for large files.

Shared Pages

Multiple processes mapping the same file share physical memory pages.

OS Caching

The OS manages page eviction and prefetching automatically.

Design Rationale

Why Memory Mapping?

Performance: Critical for streaming large game assets (textures, models, audio)
Simplicity: Reduces abstraction overhead - files are just byte slices
Integration: Works seamlessly with zero-copy parsing and deserialization

Current Limitations

No Write Support

The current implementation is read-only. Write operations would require:

MmapMut for mutable mappings
Synchronization primitives for concurrent access
Flush semantics for durability

No Virtual Filesystem

Missing features for a complete VFS:

Directory enumeration
File metadata (size, timestamps)
Mount point abstraction
Archive support (ZIP, NACP)

No Async I/O

The API is synchronous. Async support would enable:

Non-blocking file opens
Concurrent loading of multiple files
Better integration with async game loops

Future Expansion

Planned VFS Features

// Future API sketch (not implemented)

pub trait FileSystem {
    fn open(&self, path: &str) -> Result<Box<dyn VFile>>;
    fn read_dir(&self, path: &str) -> Result<Vec<DirEntry>>;
    fn metadata(&self, path: &str) -> Result<Metadata>;
}

pub trait VFile: Read + Seek {
    fn size(&self) -> u64;
    fn mmap(&self) -> Result<Mmap>;
}

// Backend implementations
pub struct HostFs;    // Real filesystem
pub struct RomFs;     // Nintendo RomFS archives
pub struct UnionFs;   // Layered filesystems (mods)

Archive Support

Nintendo Switch games use several archive formats:

NSP (Nintendo Submission Package) - PFS0 container
XCI (NX Card Image) - HFS0 cartridge image
NCA (Nintendo Content Archive) - Encrypted content
RomFS - Read-only filesystem

Supporting these requires:

pub struct NspArchive {
    base: File,
    entries: Vec<PfsEntry>,
}

impl FileSystem for NspArchive {
    fn open(&self, path: &str) -> Result<Box<dyn VFile>> {
        // Lookup path in archive entries
        // Return virtual file backed by base file slice
    }
}

Thread Safety

The current File type is not Sync because:

Mmap is !Sync by default
Multiple threads reading the same mapping is safe
But we don’t wrap it in Arc<Mmap> currently

For multi-threaded access:

use std::sync::Arc;

let file = Arc::new(File::open("data.bin")?);
let file_clone = Arc::clone(&file);

std::thread::spawn(move || {
    // Use file_clone
});

Error Handling

File operations can fail in several ways:

File Not Found
Permission Denied
Mmap Failure

match File::open("missing.bin") {
    Ok(file) => { /* use file */ },
    Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
        eprintln!("File does not exist");
    },
    Err(e) => return Err(e),
}

match File::open("/root/secret.bin") {
    Ok(file) => { /* use file */ },
    Err(e) if e.kind() == std::io::ErrorKind::PermissionDenied => {
        eprintln!("Access denied");
    },
    Err(e) => return Err(e),
}

// Mapping can fail if:
// - File is on a network drive
// - File is locked by another process
// - Not enough address space (32-bit systems)
match File::open("huge.bin") {
    Ok(file) => { /* use file */ },
    Err(e) => {
        eprintln!("Failed to map file: {}", e);
    },
}

Performance Considerations

Memory Usage

Memory-mapped files don’t consume RAM immediately:

let big_file = File::open("10GB.bin")?;  // Fast - doesn't load file
let first_mb = &big_file[0..1024*1024];  // Only this 1MB is paged in

Page Faults

First access to each 4KB page triggers a minor page fault (~1μs overhead):

// Sequential access - good prefetching
for chunk in file.chunks(4096) {
    process(chunk);  // OS prefetches next pages
}

// Random access - poor locality
for i in (0..file.len()).step_by(1_000_000) {
    let byte = file[i];  // Each access likely a page fault
}

Unmapping

Files are automatically unmapped when File is dropped:

{
    let file = File::open("temp.bin")?;
    // use file
}  // <- file unmapped here, pages freed

For long-running processes, consider manual memory management:

let file = File::open("data.bin")?;
let data = file.to_vec();  // Copy to owned Vec
drop(file);                // Unmap immediately

Integration with Other Systems

ROM Loader

use oboromi_core::fs::File;

pub struct RomLoader {
    nca_files: Vec<File>,
}

impl RomLoader {
    pub fn load(rom_path: &str) -> Result<Self, std::io::Error> {
        let nsp = File::open(rom_path)?;
        // Parse NSP header, extract NCA files
        let nca_files = extract_ncas(&nsp)?;
        Ok(Self { nca_files })
    }
}

Asset Streaming

pub struct AssetManager {
    romfs: File,
    index: HashMap<String, (u64, u64)>,  // path -> (offset, size)
}

impl AssetManager {
    pub fn load_asset(&self, path: &str) -> Option<&[u8]> {
        let (offset, size) = self.index.get(path)?;
        let start = *offset as usize;
        let end = start + *size as usize;
        Some(&self.romfs[start..end])
    }
}

Testing

#[cfg(test)]
mod tests {
    use super::*;
    use std::io::Write;
    
    #[test]
    fn test_file_open() {
        // Create temporary file
        let mut temp = std::fs::File::create("/tmp/test.bin").unwrap();
        temp.write_all(b"Hello, World!").unwrap();
        drop(temp);
        
        // Open with VFS
        let file = File::open("/tmp/test.bin").unwrap();
        assert_eq!(file.len(), 13);
        assert_eq!(&file[0..5], b"Hello");
        
        // Cleanup
        std::fs::remove_file("/tmp/test.bin").unwrap();
    }
}

ROM Loader - Game cartridge loading
Archive Support - NSP/XCI/NCA parsing
Save Data - User data management

Source Files

Implementation: core/src/fs/mod.rs:1-26
Module: core/src/lib.rs:2

Get Started

Architecture

Core Components

Development

Overview

Architecture

File Interface

Opening Files

Deref Implementation

Usage Examples

Loading a ROM File

Parsing File Structures

Memory Mapping Benefits

Zero-Copy

Lazy Loading

Shared Pages

OS Caching

Design Rationale

Why Memory Mapping?

Current Limitations

Future Expansion

Planned VFS Features

Archive Support

Thread Safety

Error Handling

Performance Considerations

Memory Usage

Page Faults

Unmapping

Integration with Other Systems

ROM Loader

Asset Streaming

Testing

Source Files

Build docs developers (and LLMs) love

Get Started

Architecture

Core Components

Development

​Overview

​Architecture

​File Interface

​Opening Files

​Deref Implementation

​Usage Examples

​Loading a ROM File

​Parsing File Structures

​Memory Mapping Benefits

Zero-Copy

Lazy Loading

Shared Pages

OS Caching

​Design Rationale

​Why Memory Mapping?

​Current Limitations

​Future Expansion

​Planned VFS Features

​Archive Support

​Thread Safety

​Error Handling

​Performance Considerations

​Memory Usage

​Page Faults

​Unmapping

​Integration with Other Systems

​ROM Loader

​Asset Streaming

​Testing

​Related Components

​Source Files

Build docs developers (and LLMs) love

Overview

Architecture

File Interface

Opening Files

Deref Implementation

Usage Examples

Loading a ROM File

Parsing File Structures

Memory Mapping Benefits

Design Rationale

Why Memory Mapping?

Current Limitations

Future Expansion

Planned VFS Features

Archive Support

Thread Safety

Error Handling

Performance Considerations

Memory Usage

Page Faults

Unmapping

Integration with Other Systems

ROM Loader

Asset Streaming

Testing

Related Components

Source Files