Yjs Integration

Ganimede uses Yjs (Conflict-free Replicated Data Types) for real-time synchronization between the backend and all connected clients. This enables collaborative editing and consistent state across all instances.

Why Yjs?

Yjs provides:

Automatic conflict resolution: Multiple users can edit simultaneously without conflicts
Offline support: Changes can be made offline and synced later
Efficient sync: Only deltas are transmitted over the network
Strong eventual consistency: All clients converge to the same state
Undo/redo: Built-in transaction history

Architecture Overview

Frontend (y-websocket)     Backend (ypy-websocket)
        │                           │
        ▼                           ▼
    YDoc (JS) ◄────WebSocket────► YDoc (Python)
        │                           │
        ▼                           ▼
  Svelte stores              Managers (Notebook, Kernel)

Both frontend and backend share the same YDoc structure, synchronized via WebSocket.

YDoc Structure

The YDoc contains all notebook state using Yjs shared types:

// Top-level shared types
ydoc.getArray("cells")        // [cell_id1, cell_id2, ...]
ydoc.getMap("np_graph")        // { cell_id: [next_ids] }
ydoc.getMap("pc_graph")        // { parent_id: [child_ids] }
ydoc.getMap("kernel")          // { busy: false }
ydoc.getArray("run_queue")     // [cell_id1, cell_id2, ...]
ydoc.getText("nb_path")        // "/path/to/notebook.ipynb"

// Per-cell maps
ydoc.getMap(cell_id) = {
  id: string,
  type: "code" | "markdown",
  source: YText,              // Collaborative text editing
  execution_count: number | null,
  outputs: YArray,            // [output1, output2, ...]
  top: number,                // Y position on canvas
  left: number,               // X position on canvas
  width: number,              // Cell width in px
  height: number,             // Cell height in px
  collapsed: string | null,   // "i", "o", "b", "h"
  state: "idle" | "queued" | "running"
}

Shared Types

YArray

Ordered list with insert/delete operations:

// Frontend
const ycells = ydoc.getArray("cells");
ycells.push(["new_cell_id"]);
ycells.delete(0, 1);
const cells = ycells.toJSON();

# Backend
ycells = ydoc.get_array("cells")
with ydoc.begin_transaction() as t:
    ycells.append(t, "new_cell_id")
    ycells.delete(t, 0)

YMap

Key-value map with set/delete operations:

// Frontend
const ycell = ydoc.getMap(cell_id);
ycell.set("state", "running");
const state = ycell.get("state");

# Backend
ycell = ydoc.get_map(cell_id)
with ydoc.begin_transaction() as t:
    ycell.set(t, "state", "running")
state = ycell.get("state")

YText

Collaborative text editing with character-level operations:

// Frontend
const source = ydoc.getMap(cell_id).get("source");
source.insert(0, "print('hello')");
source.delete(0, 5);
const text = source.toString();

# Backend
source = Y.YText(("print('hello')"))
with ydoc.begin_transaction() as t:
    ycell.set(t, "source", source)

YText is bound to Monaco editor for collaborative editing:

import { MonacoBinding } from "y-monaco";

const binding = new MonacoBinding(
  ycell.get("source"),  // YText
  editor.getModel(),
  new Set([editor]),
  awareness
);

Transactions

All YDoc mutations must happen within a transaction. This ensures atomicity and efficient sync. Backend (Python):

with ydoc.begin_transaction() as t:
    ycell.set(t, "state", "running")
    ycell.set(t, "execution_count", 5)
    # Both changes sent as a single update

Frontend (JavaScript):

// Automatic transaction
ycell.set("state", "idle");

// Manual transaction (for batching)
ydoc.transact(() => {
  ycell.set("state", "idle");
  ycell.set("execution_count", 5);
});

WebSocket Providers

Frontend: y-websocket

Location: ui/src/stores/_notebook.js

import { WebsocketProvider } from "y-websocket";

const ydoc = new Y.Doc();
const websocket_provider = new WebsocketProvider(
  "ws://localhost:1234",
  "g-y-room",
  ydoc
);

websocket_provider.on("status", event => {
  console.log(event.status);  // "connected" or "disconnected"
});

Backend: ypy-websocket

Location: api/ganimede/main.py

from ypy_websocket import WebsocketServer, WebsocketProvider
import y_py as Y
from websockets import serve

# Create server
websocket_server = WebsocketServer()
ydoc = Y.YDoc()

# Start server task
async def server_task():
    async with WebsocketServer(log=log) as websocket_server:
        async with serve(websocket_server.serve, "localhost", 1234):
            await asyncio.Future()  # run forever

# Connect as client (for backend access to YDoc)
websocket = await connect("ws://localhost:1234/g-y-room")
websocket_provider = WebsocketProvider(ydoc, websocket)
await websocket_provider.start()

Observation Patterns

Yjs provides observers to react to changes.

Frontend (JavaScript)

// Observe array changes
ycells.observe((event) => {
  console.log("Added:", event.changes.added);
  console.log("Deleted:", event.changes.deleted);
  cells = ycells.toJSON();  // Trigger Svelte reactivity
});

// Observe map changes
ycell.observe((event) => {
  console.log("Changed keys:", event.keysChanged);
  cell = cell;  // Force Svelte reactivity
});

// Observe deep changes (nested structures)
ynp_graph.observeDeep((event) => {
  np_graph.set(ynp_graph.toJSON());
});

Integration with Svelte:

let cell = {
  ycell: ydoc.getMap(cell_id),
  
  get state() { return this.ycell.get("state"); },
  set state(value) { this.ycell.set("state", value); }
};

// Force Svelte to react to YDoc changes
cell.ycell.observe((yevent) => {
  cell = cell;  // Trigger reassignment
});

Backend (Python)

Python Yjs doesn’t support observers in the same way. Instead, the backend updates YDoc and relies on WebSocket sync to propagate changes.

Data Flow Examples

Example 1: User Edits Cell Source

1. User types in Monaco editor
      │
      ▼
2. MonacoBinding.insert() → YText.insert()
      │
      ▼
3. YDoc transaction created
      │
      ▼
4. y-websocket provider sends update to server
      │
      ▼
5. ypy-websocket server receives update
      │
      ▼
6. Server broadcasts to all other clients
      │
      ▼
7. Other clients' y-websocket providers receive update
      │
      ▼
8. YDoc.applyUpdate() applies changes
      │
      ▼
9. MonacoBinding updates other users' editors

Example 2: Backend Updates Cell State

1. Kernel starts executing
      │
      ▼
2. Notebook._change_cell_state(cell_id, "running")
      │
      ▼
3. with ydoc.begin_transaction() as t:
       ycell.set(t, "state", "running")
      │
      ▼
4. ypy-websocket server broadcasts update
      │
      ▼
5. Frontend y-websocket provider receives update
      │
      ▼
6. ycell.observe() fires
      │
      ▼
7. cell = cell  // Force Svelte reactivity
      │
      ▼
8. UI updates to show running state

Example 3: Cell Execution Output

1. Jupyter kernel sends output to IOPub
      │
      ▼
2. Kernel.execute() receives output message
      │
      ▼
3. msg_queue.put_nowait(output)
      │
      ▼
4. Notebook.run() processes output
      │
      ▼
5. with ydoc.begin_transaction() as t:
       ycell.get("outputs").append(t, output)
      │
      ▼
6. ypy-websocket broadcasts to clients
      │
      ▼
7. Frontend ycell.get("outputs").observe() fires
      │
      ▼
8. Outputs.svelte re-renders with new output

Graph Structures

The notebook uses two graph structures for cell organization:

Next-Prev Graph (np_graph)

Defines sequential execution flow:

// YMap<cell_id, YArray<next_cell_ids>>
ynp_graph = {
  "cell1": ["cell2"],
  "cell2": ["cell3"],
  "cell3": []
}

This creates the flow: cell1 → cell2 → cell3

Parent-Children Graph (pc_graph)

Defines tissue grouping (heading cells with children):

// YMap<parent_id, YArray<child_ids>>
ypc_graph = {
  "heading1": ["cell1", "cell2"],
  "heading2": ["cell3"]
}

The frontend derives the reverse mapping:

// Derived store: child → parent
const cp_graph = derived(pc_graph, $pc_graph => {
  const cp = {};
  for (const parent in $pc_graph) {
    for (const child of $pc_graph[parent]) {
      cp[child] = parent;
    }
  }
  return cp;
});

Initialization Flow

Backend (main.py:47)

@app.on_event("startup")
async def on_startup():
    # 1. Start Yjs server
    loop.create_task(server_task())
    
    # 2. Connect to Yjs server
    websocket = await connect("ws://localhost:1234/g-y-room")
    websocket_provider = WebsocketProvider(ydoc, websocket)
    await websocket_provider.start()
    
    # 3. Load notebook into YDoc
    notebook = Notebook(notebook_path, kernel, comms, ydoc)
    # notebook.init_cells() populates ydoc with cells

Frontend (_notebook.js)

// Create YDoc
export const ydoc = new Y.Doc();

// Connect to backend Yjs server
const websocket_provider = new WebsocketProvider(
  "ws://localhost:1234",
  "g-y-room",
  ydoc
);

// YDoc is now synchronized with backend
// Access shared types
export const ycells = ydoc.getArray("cells");
export const ynp_graph = ydoc.getMap("np_graph");

Undo/Redo

Frontend (_notebook.js:58):

const undoManager = new Y.UndoManager(ycells);

document.addEventListener("keydown", (event) => {
  if ((event.ctrlKey || event.metaKey) && event.key === "z") {
    undoManager.undo();
  }
});

// Stop capturing to group operations
undoManager.stopCapturing();

The UndoManager tracks changes to ycells and can undo/redo them.

Awareness Protocol

Awareness is a separate protocol for ephemeral state (cursors, selections).

export const awareness = websocket_provider.awareness;

// Set local state
awareness.setLocalStateField("id", generateId());
awareness.setLocalStateField("color", "#FF0000");
awareness.setLocalStateField("cursor", { x: 100, y: 200 });

// Observe remote state
awareness.on("change", (changes) => {
  for (let [clientId, state] of awareness.getStates()) {
    console.log(clientId, state.cursor);
  }
});

Awareness state is not part of YDoc and is not persisted. It’s only used for real-time presence.

Performance Considerations

Batch updates: Use transactions to batch multiple changes

ydoc.transact(() => {
  ycell.set("top", 100);
  ycell.set("left", 200);
  ycell.set("width", 300);
});

Observe efficiently: Use observe() instead of observeDeep() when possible

Avoid frequent toJSON(): Cache the JSON representation and only update on changes

let cells = ycells.toJSON();
ycells.observe(() => {
  cells = ycells.toJSON();  // Only when changed
});

Stop capturing for non-undoable operations:

undoManager.stopCapturing();
ycells.push(["temp_cell"]);

Conflict Resolution

Yjs automatically resolves conflicts:

YText: Character-level CRDT (similar to OT)
YArray: Position-based inserts with unique identifiers
YMap: Last-write-wins for each key

No manual conflict resolution is needed.

Key Patterns

YDoc as single source of truth: All state flows through YDoc
Transactions for atomicity: Batch related changes
Observers for reactivity: YDoc changes trigger UI updates
Derived state: Compute reverse graphs (cp_graph from pc_graph)
Separation of concerns: Yjs for state, Comms for commands
Awareness for presence: Cursors and ephemeral state

Backend Architecture - How backend updates YDoc
Frontend Architecture - How frontend observes YDoc

System Design

Yjs Integration

Yjs Integration

Why Yjs?

Architecture Overview

YDoc Structure

Shared Types

YArray

YMap

YText

Transactions

WebSocket Providers

Frontend: y-websocket

Backend: ypy-websocket

Observation Patterns

Frontend (JavaScript)

Backend (Python)

Data Flow Examples

Example 1: User Edits Cell Source

Example 2: Backend Updates Cell State

Example 3: Cell Execution Output

Graph Structures

Next-Prev Graph (np_graph)

Parent-Children Graph (pc_graph)

Initialization Flow

Backend (main.py:47)

Frontend (_notebook.js)

Undo/Redo

Awareness Protocol

Performance Considerations

Conflict Resolution

Key Patterns

Build docs developers (and LLMs) love

System Design

​Yjs Integration

​Why Yjs?

​Architecture Overview

​YDoc Structure

​Shared Types

​YArray

​YMap

​YText

​Transactions

​WebSocket Providers

​Frontend: y-websocket

​Backend: ypy-websocket

​Observation Patterns

​Frontend (JavaScript)

​Backend (Python)

​Data Flow Examples

​Example 1: User Edits Cell Source

​Example 2: Backend Updates Cell State

​Example 3: Cell Execution Output

​Graph Structures

​Next-Prev Graph (np_graph)

​Parent-Children Graph (pc_graph)

​Initialization Flow

​Backend (main.py:47)

​Frontend (_notebook.js)

​Undo/Redo

​Awareness Protocol

​Performance Considerations

​Conflict Resolution

​Key Patterns

​Related

Build docs developers (and LLMs) love

Yjs Integration

Why Yjs?

Architecture Overview

YDoc Structure

Shared Types

YArray

YMap

YText

Transactions

WebSocket Providers

Frontend: y-websocket

Backend: ypy-websocket

Observation Patterns

Frontend (JavaScript)

Backend (Python)

Data Flow Examples

Example 1: User Edits Cell Source

Example 2: Backend Updates Cell State

Example 3: Cell Execution Output

Graph Structures

Next-Prev Graph (np_graph)

Parent-Children Graph (pc_graph)

Initialization Flow

Backend (main.py:47)

Frontend (_notebook.js)

Undo/Redo

Awareness Protocol

Performance Considerations

Conflict Resolution

Key Patterns

Related