Storage System

Opal Editor’s storage system provides a unified abstraction over multiple storage backends through the Disk class. This enables seamless switching between IndexedDB, Origin Private File System (OPFS), in-memory storage, and local directory mounting.

Architecture

┌─────────────────────────────────────────┐
│              Disk                       │
│    (Abstract Storage Interface)         │
├─────────────────────────────────────────┤
│  DiskContext (fs + fileTree + mutex)    │
├─────────────────────────────────────────┤
│  ┌──────────┬──────────┬──────────┐     │
│  │ IndexedDB│  OPFS    │  Memory  │     │
│  │   Disk   │   Disk   │   Disk   │     │
│  └──────────┴──────────┴──────────┘     │
└─────────────────────────────────────────┘
        │           │           │
        ▼           ▼           ▼
   Lightning    OPFS API    In-Memory
      FS        (memfs)      (memfs)

Core Class: Disk

Location: ~/workspace/source/src/data/disk/Disk.ts

Abstract Base Class

export abstract class Disk<TContext extends DiskContext = DiskContext> {
  readonly guid: string;
  abstract type: DiskType;
  
  // Storage interface
  get fs(): CommonFileSystem;
  
  // File tree cache
  get fileTree(): FileTree;
  
  // Concurrency control
  mutex: Mutex;
  
  // Event system
  local: DiskEventsLocal;
  remote: DiskEventsRemote;
  
  // Initialization promise
  ready: Promise<void>;
}

Disk Types

type DiskType = 
  | "IndexedDbDisk"      // Persistent browser storage
  | "OpFsDisk"           // Origin Private File System
  | "OpFsDirMountDisk"   // Mount local directory (File System Access API)
  | "MemDisk"            // In-memory storage
  | "NullDisk";          // No-op implementation

Storage Backends

IndexedDB Disk

Location: ~/workspace/source/src/data/disk/IndexedDbDisk.ts Persistent storage using @isomorphic-git/lightning-fs.

import { IndexedDbDisk } from "@/data/disk/IndexedDbDisk";

// Create IndexedDB disk
const disk = new IndexedDbDisk("my-disk-guid");
await disk.init();

// Files persist across sessions
await disk.writeFile(absPath("/file.txt"), "content");

Characteristics:

✅ Persistent across browser sessions
✅ Works in all modern browsers
✅ No user permissions required
⚠️ Storage quotas apply (~50-100MB typical)
⚠️ Slower than OPFS

OPFS Directory Mount Disk

Location: ~/workspace/source/src/data/disk/OPFsDirMountDisk.ts Mount a local directory using the File System Access API.

import { OpFsDirMountDisk } from "@/data/disk/OPFsDirMountDisk";

// Create and select directory
const disk = new OpFsDirMountDisk("my-disk-guid");
const handle = await disk.selectDirectory();
await disk.init();

// Or create with directory upfront
const handle = await window.showDirectoryPicker();
const disk = await OpFsDirMountDisk.CreateWithDirectory("guid");

// Access directory name
console.log(disk.dirName); // Directory name

// Check if selection needed
if (await disk.needsDirectorySelection()) {
  await disk.selectDirectory();
}

Characteristics:

✅ Real file system access
✅ Survives browser restarts (after re-permission)
✅ Can access existing folders
✅ Changes visible to other apps
⚠️ Requires user permission
⚠️ Handle persistence requires re-verification

Directory Handle Management:

// Handles are stored automatically
await disk.setDirectoryHandle(handle);

// Check if has handle
if (disk.hasDirectoryHandle()) {
  console.log(disk.getDirectoryName());
}

// Get stored metadata
const metadata = await disk.getStoredMetadata();

Memory Disk

In-memory storage for temporary operations.

import { MemDisk } from "@/data/disk/MemDisk";

const disk = new MemDisk("temp-guid");
await disk.init();

// Data lost on page reload
await disk.writeFile(absPath("/temp.txt"), "data");

Characteristics:

✅ Very fast
✅ No storage quotas
✅ Useful for previews/temp files
❌ Data lost on page reload

File Operations

Reading and Writing

// Write file
await disk.writeFile(absPath("/file.txt"), "content");
await disk.writeFile(absPath("/data.json"), JSON.stringify(data));
await disk.writeFile(absPath("/image.png"), uint8Array);

// Read file
const content = await disk.readFile(absPath("/file.txt"));
const text = String(content);

// Write with recursive directory creation
await disk.writeFileRecursive(
  absPath("/deep/nested/file.txt"),
  "content"
);

// Check if path exists
const exists = await disk.pathExists(absPath("/file.txt"));

Creating Files and Directories

// Create single file (auto-increments if exists)
const path = await disk.newFile(
  absPath("/file.txt"),
  "content"
);

// Create multiple files
const paths = await disk.newFiles([
  [absPath("/file1.txt"), "content 1"],
  [absPath("/file2.txt"), "content 2"],
  [absPath("/file3.txt"), Promise.resolve("async content")]
]);

// Create directory
const dirPath = await disk.newDir(absPath("/folder"));

// Create directory recursively
await disk.mkdirRecursive(absPath("/deep/nested/folder"));

Renaming and Moving

// Rename file
const result = await disk.renameFile(
  absPath("/old.txt"),
  absPath("/new.txt")
);

// Rename directory
await disk.renameDir(
  absPath("/old-folder"),
  absPath("/new-folder")
);

// Rename multiple (atomic operation)
const results = await disk.renameMultiple([
  [node1, absPath("/new-path-1.txt")],
  [node2, absPath("/new-path-2.txt")]
]);

// Quiet move (no events)
const finalPath = await disk.quietMove(
  absPath("/source.txt"),
  absPath("/dest.txt"),
  { overWrite: true }
);

Copying

// Copy file
await disk.copyFile(
  absPath("/source.txt"),
  absPath("/dest.txt"),
  false // overwrite
);

// Copy directory
await disk.copyDir(
  absPath("/source-folder"),
  absPath("/dest-folder")
);

// Copy multiple
const paths = await disk.copyMultiple([
  [sourceNode1, absPath("/dest1.txt")],
  [sourceNode2, absPath("/dest2.txt")]
]);

// Copy from another disk
await disk.copyMultipleSourceNodes(
  sourceNodes,
  sourceDisk
);

// Copy entire disk
await disk.copyDiskToDisk(targetDisk);

Deleting

// Remove single file
await disk.removeFile(absPath("/file.txt"));

// Remove multiple files
await disk.removeMultipleFiles([
  absPath("/file1.txt"),
  absPath("/file2.txt")
]);

File Tree System

Accessing the File Tree

// Get file tree root
const root = disk.fileTree.root;

// Get node by path
const node = disk.nodeFromPath(absPath("/file.txt"));

if (node?.isTreeFile()) {
  console.log("File:", node.path);
} else if (node?.isTreeDir()) {
  console.log("Directory:", node.path);
}

// Get first file
const firstFile = disk.getFirstFile();

Filtering and Iteration

// Get flat tree with filters
const markdownFiles = disk.getFlatTree({
  filterIn: (node) => node.isMarkdownFile(),
  filterOut: (node) => node.path.startsWith("/.trash")
});

// Iterate with generator
for await (const { filePath, text } of disk.scan()) {
  console.log(`${filePath}: ${text}`);
}

Virtual Files

Virtual files exist in the tree but not on disk.

// Add virtual file
const virtualNode = disk.addVirtualFile({
  type: "file",
  basename: "temp.md",
  selectedNode: parentNode,
  virtualContent: async () => "Generated content",
  source: sourceNode // optional
});

// Remove virtual file
disk.removeVirtualFile(absPath("/temp.md"));

Indexing System

File Tree Indexing

// Trigger re-index
await disk.triggerIndex();

// Superficial index (no cache write)
await disk.superficialIndex();

// Wait for first index
await disk.tryFirstIndex();

Index Listeners

// Listen to latest index
const unsub = disk.latestIndexListener((fileTree, trigger) => {
  console.log("Files:", fileTree.root.children);
  
  if (trigger?.type === "create") {
    console.log("Created:", trigger.details.filePaths);
  }
});

// Specific event listeners
disk.createListener(({ filePaths }) => {
  console.log("Created:", filePaths);
});

disk.renameListener((changes) => {
  changes.forEach(({ oldPath, newPath }) => {
    console.log(`Renamed: ${oldPath} → ${newPath}`);
  });
});

disk.deleteListener(({ filePaths }) => {
  console.log("Deleted:", filePaths);
});

// Clean up
unsub();

Event System

Disks use both local and remote events for cross-tab synchronization.

Event Types

const DiskEvents = {
  INDEX: "index",           // File tree changed
  INSIDE_WRITE: "inside-write",   // File written (no watchers)
  OUTSIDE_WRITE: "outside-write", // File written (trigger watchers)
};

Write Events

// Listen to outside writes (user edits)
disk.outsideWriteListener(absPath("/config.json"), (contents) => {
  console.log("Config updated:", contents);
});

// Listen to inside writes (programmatic)
disk.insideWriteListener(absPath("/cache.json"), (contents) => {
  console.log("Cache updated:", contents);
});

// Listen to any write or index change
disk.dirtyListener((trigger) => {
  console.log("Disk modified");
});

// Listen to index updates
disk.writeIndexListener(() => {
  console.log("Index or write occurred");
});

Advanced Operations

Batch Find-Replace

// Replace image URLs in markdown files
const modifiedFiles = await disk.findReplaceImgBatch(
  [
    ["old-image.png", "new-image.png"],
    ["logo.jpg", "brand-logo.jpg"]
  ],
  "https://example.com" // origin URL
);

// Replace file links in markdown files
const modifiedFiles = await disk.findReplaceFileBatch(
  [["old-page.md", "new-page.md"]],
  "/blog"
);

Path Utilities

// Get next available path (handles conflicts)
const uniquePath = await disk.nextPath(absPath("/file.txt"));
// If /file.txt exists, returns /file (2).txt

Initialization and Lifecycle

Initialize Disk

const disk = new IndexedDbDisk("my-guid");

// Full initialization with listeners
const cleanup = await disk.init();

// Or skip listeners (for workers)
await disk.init({ skipListeners: true });

// Refresh from storage
await disk.refresh();

Cleanup

// Tear down (cleanup without deletion)
await disk.tearDown();

// Destroy (delete all data)
await disk.destroy();

Serialization

// To JSON
const json = disk.toJSON();
// Returns: { type, guid, indexCache? }

// From JSON
const disk = DiskFromJSON(json);

Factory Pattern

import { DiskFromJSON, DiskFactoryByType } from "@/data/disk/DiskFactory";

// Create from type
const disk = DiskFactoryByType("IndexedDbDisk");

// Create from JSON
const disk = DiskFromJSON({
  type: "IndexedDbDisk",
  guid: "my-guid",
  indexCache: cachedTree
});

Context Pattern

Disk Context

Each disk type has an associated context that manages:

File system interface
File tree instance
Mutex for concurrency

class DiskContext {
  constructor(
    public fs: CommonFileSystem,
    public fileTree: FileTree,
    public mutex: Mutex
  ) {}
  
  abstract tearDown(): Promise<void>;
}

Switching Contexts

// Advanced: switch disk context
const newContext = IndexedDbDiskContext.create(guid, indexCache);
await disk.setDiskContext(newContext);

Database Layer

Location: ~/workspace/source/src/data/db/DBInstance.ts Disks use IndexedDB for metadata persistence through the ClientDb singleton:

import { ClientDb } from "@/data/db/DBInstance";

// ClientDb is a lazy-initialized proxy to ClientIndexedDb
// Accessed automatically by DiskDAO for persistence

Best Practices

Always Initialize

// ✅ Good
const disk = new IndexedDbDisk("guid");
await disk.init();
await disk.ready;
await disk.writeFile(absPath("/file.txt"), "content");

// ❌ Bad - may fail
const disk = new IndexedDbDisk("guid");
await disk.writeFile(absPath("/file.txt"), "content");

Use Path Helpers

import { absPath, relPath, joinPath } from "@/lib/paths2";

// ✅ Good - type-safe
await disk.writeFile(absPath("/file.txt"), content);

// ❌ Bad - no type safety
await disk.writeFile("/file.txt", content);

Handle Index Events

// Wait for initial index before operations
await disk.tryFirstIndex();

// Listen for changes
disk.latestIndexListener((tree) => {
  // Tree is always up-to-date
  updateUI(tree);
});

Cleanup Resources

try {
  const disk = new IndexedDbDisk("guid");
  const cleanup = await disk.init();
  // ... operations
} finally {
  cleanup(); // Remove event listeners
  await disk.tearDown();
}

Use Quiet Operations for Performance

// ✅ Good - no events during batch
for (const file of files) {
  await disk.newFileQuiet(file.path, file.content);
}
await disk.triggerIndex(); // Single re-index
voiddisk.local.emit(DiskEvents.INDEX, { type: "create", details });

// ❌ Bad - triggers index per file
for (const file of files) {
  await disk.newFile(file.path, file.content);
}

Common Patterns

Local Directory Workspace

const disk = new OpFsDirMountDisk("workspace-guid");
if (await disk.needsDirectorySelection()) {
  await disk.selectDirectory();
}
await disk.init();

Temp Disk for Preview

const tempDisk = new MemDisk("preview-" + nanoid());
await tempDisk.init({ skipListeners: true });
// ... generate preview files
await tempDisk.tearDown();

Cross-Disk Copy

await sourceDisk.copyDiskToDisk(targetDisk);

Error Handling

import { NotFoundError } from "@/lib/errors/errors";

try {
  await disk.readFile(absPath("/missing.txt"));
} catch (error) {
  if (error instanceof NotFoundError) {
    console.log("File not found");
  }
}

Workspace System - Uses disk for file storage
Git System - Reads/writes via disk.fs
Build System - Reads source from disk

Getting Started

Core Systems

Contributing

​Storage System

​Architecture

​Core Class: Disk

​Abstract Base Class

​Disk Types

​Storage Backends

​IndexedDB Disk

​OPFS Directory Mount Disk

​Memory Disk

​File Operations

​Reading and Writing

​Creating Files and Directories

​Renaming and Moving

​Copying

​Deleting

​File Tree System

​Accessing the File Tree

​Filtering and Iteration

​Virtual Files

​Indexing System

​File Tree Indexing

​Index Listeners

​Event System

​Event Types

​Write Events

​Advanced Operations

​Batch Find-Replace

​Path Utilities

​Initialization and Lifecycle

​Initialize Disk

​Cleanup

​Serialization

​Factory Pattern

​Context Pattern

​Disk Context

​Switching Contexts

​Database Layer

​Best Practices

​Always Initialize

​Use Path Helpers

​Handle Index Events

​Cleanup Resources

​Use Quiet Operations for Performance

​Common Patterns

​Local Directory Workspace

​Temp Disk for Preview

​Cross-Disk Copy

​Error Handling

​Related Systems

Build docs developers (and LLMs) love

Storage System

Architecture

Core Class: Disk

Abstract Base Class

Disk Types

Storage Backends

IndexedDB Disk

OPFS Directory Mount Disk

Memory Disk

File Operations

Reading and Writing

Creating Files and Directories

Renaming and Moving

Copying

Deleting

File Tree System

Accessing the File Tree

Filtering and Iteration

Virtual Files

Indexing System

File Tree Indexing

Index Listeners

Event System

Event Types

Write Events

Advanced Operations

Batch Find-Replace

Path Utilities

Initialization and Lifecycle

Initialize Disk

Cleanup

Serialization

Factory Pattern

Context Pattern

Disk Context

Switching Contexts

Database Layer

Best Practices

Always Initialize

Use Path Helpers

Handle Index Events

Cleanup Resources

Use Quiet Operations for Performance

Common Patterns

Local Directory Workspace

Temp Disk for Preview

Cross-Disk Copy

Error Handling

Related Systems