Mesophyll Coordination Layer

Mesophyll is the coordination layer that manages communication between the master process and worker processes in Template Worker. It provides a WebSocket-based protocol for real-time bidirectional communication, replacing the older HTTP/2-based WorkerProcessComm system.

Overview

Mesophyll runs on the master process and maintains WebSocket connections to all worker processes. It enables:

Event dispatching from master to workers
Response collection from workers back to master
Heartbeat monitoring for connection health
Database state synchronization
Template shop updates across all workers

Mesophyll is currently implemented as part of the master process. Future plans include using it as a base unit for sandboxing through projects like Khronos DAPI.

Architecture

The Mesophyll system consists of two main components:

Server Component

The MesophyllServer runs on the master process and:

Listens for WebSocket connections from workers (src/mesophyll/server.rs:42-43)
Authenticates workers using token-based authentication (src/mesophyll/server.rs:86-96)
Manages active connections in a concurrent map (src/mesophyll/server.rs:18)
Provides HTTP endpoints for database operations (src/mesophyll/server.rs:44-48)

pub struct MesophyllServer {
    idents: Arc<HashMap<usize, String>>,
    conns: Arc<DashMap<usize, MesophyllServerConn>>,
    db_state: DbState,
}

Client Component

The MesophyllClient runs in each worker process and:

Establishes WebSocket connection to the master (src/mesophyll/client.rs:45)
Sends periodic heartbeats to maintain connection (src/mesophyll/client.rs:115-118)
Receives and processes server messages (src/mesophyll/client.rs:52-94)
Automatically reconnects on connection loss (src/mesophyll/client.rs:29-38)

pub struct MesophyllClient {
    wt: Arc<WorkerThread>,
    addr: String,
}

WebSocket Protocol

Connection Establishment

Workers connect to the Mesophyll server using WebSocket:

ws://{server_addr}/ws?id={worker_id}&token={auth_token}

Authentication: Each worker has a unique 64-character alphanumeric token generated by the server (src/mesophyll/server.rs:23-29). The server validates both the worker ID and token before upgrading to WebSocket (src/mesophyll/server.rs:86-99).

Message Format

All messages are serialized using MessagePack (rmp_serde) and sent as binary WebSocket frames (src/mesophyll/server.rs:524-538).

fn encode_message<T: serde::Serialize>(msg: &T) -> Result<Message, crate::Error> {
    let bytes = rmp_serde::encode::to_vec(msg)?;
    Ok(Message::Binary(bytes.into()))
}

fn decode_message<T: for<'de> serde::Deserialize<'de>>(msg: &Message) -> Result<T, crate::Error> {
    match msg {
        Message::Binary(b) => {
            rmp_serde::from_slice(b)
        }
        _ => Err("Invalid Mesophyll message type".into()),
    }
}

Message Types

Server Messages

Messages sent from server to client (src/mesophyll/message.rs:6-33):

Hello

Sent after connection establishment to configure the heartbeat interval:

ServerMessage::Hello { 
    heartbeat_interval_ms: u64 
}

DispatchEvent

Dispatches a custom template event to a worker:

ServerMessage::DispatchEvent { 
    id: Id,                    // Tenant identifier (guild, user, etc.)
    event: CreateEvent,        // Event to dispatch
    req_id: Option<u64>        // Request ID for response correlation
}

If req_id is Some, the worker must send a DispatchResponse (src/mesophyll/server.rs:472-481)
If req_id is None, it’s a fire-and-forget dispatch (src/mesophyll/server.rs:496-500)

RunScript

Executes arbitrary Luau code in a worker (useful for internal tooling):

ServerMessage::RunScript { 
    id: Id,
    name: String,              // Chunk name for debugging
    code: String,              // Luau code to execute
    event: CreateEvent,        // Event context
    req_id: u64                // Required request ID
}

DropWorker

Requests a worker to drop a tenant VM:

ServerMessage::DropWorker { 
    id: Id,                    // Tenant to drop
    req_id: u64                // Required request ID
}

Client Messages

Messages sent from client to server (src/mesophyll/message.rs:35-45):

DispatchResponse

Response to a DispatchEvent or RunScript request:

ClientMessage::DispatchResponse { 
    req_id: u64,                           // Matches the request ID
    result: Result<KhronosValue, String>   // Execution result or error
}

Heartbeat

Periodic heartbeat to keep connection alive:

ClientMessage::Heartbeat { }

Sent at the interval specified in the Hello message (default 5000ms) (src/mesophyll/mod.rs:18).

Request/Response Pattern

Mesophyll uses a request ID correlation system for async request/response:

Server generates a random u64 request ID (src/mesophyll/server.rs:457-464)
Server registers a oneshot channel handler for the request ID
Server sends the message with the request ID
Client processes the message and sends back a DispatchResponse with the same request ID
Server correlates the response using the request ID (src/mesophyll/server.rs:427-430)
The awaiting task receives the response through the oneshot channel

// Server-side example from src/mesophyll/server.rs:472-481
pub async fn dispatch_event(&self, id: Id, event: CreateEvent) -> Result<KhronosValue, crate::Error> {
    let (req_id, rx) = self.register_dispatch_response_handler();
    let _guard = DispatchHandlerDropGuard::new(&self.dispatch_response_handlers, req_id);
    
    let message = ServerMessage::DispatchEvent { id, event, req_id: Some(req_id) };
    self.send(&message)?;
    Ok(rx.await??)
}

Database HTTP Endpoints

In addition to WebSocket communication, Mesophyll provides HTTP/2 endpoints for database operations (src/mesophyll/server.rs:44-48):

Tenant State Endpoints

GET /db/tenant-states - List all tenant states for a worker (src/mesophyll/server.rs:103-112)
POST /db/tenant-state - Set tenant state for a specific tenant (src/mesophyll/server.rs:135-161)

Key-Value Endpoints

POST /db/kv - Perform key-value operations (Get, Set, Delete, Find, ListScopes) (src/mesophyll/server.rs:164-230)
POST /db/public-global-kv - Public global KV operations (Find, Get) (src/mesophyll/server.rs:233-267)
POST /db/global-kv - Global KV operations (Create, Delete) (src/mesophyll/server.rs:270-309)

Authentication

All HTTP endpoints require query parameters:

id - Worker ID
token - Authentication token
tenant_type & tenant_id - For tenant-specific operations

#[derive(serde::Deserialize)]
pub struct WorkerQuery {
    id: usize,
    token: String,
}

Request/Response Format

All HTTP requests and responses use MessagePack serialization (src/mesophyll/server.rs:541-558).

Database Client

The MesophyllDbClient provides a high-level interface for workers to access database operations (src/mesophyll/client.rs:126-323):

pub struct MesophyllDbClient {
    addr: String,
    worker_id: usize,
    token: String,
    client: reqwest::Client,  // HTTP/2 enabled
}

Example Usage

// Get tenant states
let states = db_client.list_tenant_states().await?;

// Set a key-value pair
db_client.kv_set(id, scopes, key, value).await?;

// Find keys with prefix
let records = db_client.kv_find(id, scopes, prefix).await?;

The client automatically:

Uses HTTP/2 with prior knowledge (src/mesophyll/client.rs:141-144)
Constructs URLs with authentication parameters (src/mesophyll/client.rs:148-158)
Encodes requests and decodes responses using MessagePack (src/mesophyll/client.rs:173-192)

Connection Management

Heartbeat System

Clients send heartbeats at regular intervals to maintain connection health:

// Default heartbeat interval: 5 seconds
pub(super) const MESOPHYLL_DEFAULT_HEARTBEAT_MS: u64 = 5000;

The server tracks the last heartbeat timestamp for each connection (src/mesophyll/server.rs:401, 432-434).

Automatic Reconnection

Clients automatically reconnect on connection failure:

loop {
    if let Err(e) = self_ref.handle_task().await {
        log::error!("Mesophyll client task error: {}", e);
    }
    
    log::debug!("Mesophyll client reconnecting in 5 seconds...");
    tokio::time::sleep(Duration::from_secs(5)).await;
}

Reconnection Handling

When a worker reconnects, the server automatically replaces the old connection (src/mesophyll/server.rs:328-330):

if state.conns.contains_key(&id) {
    log::debug!("Worker {id} reconnection - overwriting old connection.");
}

Concurrency

Mesophyll handles multiple concurrent operations:

Dispatches: Uses FuturesUnordered to process multiple event dispatches concurrently (src/mesophyll/client.rs:48-49)
Script runs: Separate FuturesUnordered for script executions (src/mesophyll/client.rs:49)
Connection map: Uses DashMap for lock-free concurrent access to connections (src/mesophyll/server.rs:18)

Error Handling

Connection Errors

Invalid worker ID returns 404 NOT_FOUND (src/mesophyll/server.rs:87-90)
Invalid token returns 401 UNAUTHORIZED (src/mesophyll/server.rs:92-96)
Connection failures trigger automatic reconnection on client side

Message Processing

Failed message deserialization is logged but doesn’t close the connection (src/mesophyll/server.rs:422-424)
Unmatched request IDs are silently ignored (src/mesophyll/server.rs:428-430)

Database Operations

Operation failures return 500 INTERNAL_SERVER_ERROR with error message
Invalid request bodies return 400 BAD_REQUEST

Transport Layer

Mesophyll also provides a generic RPC transport abstraction (src/mesophyll/transport.rs) for future extensibility:

define_rpc_endpoints! {
    heartbeat: Heartbeat(HeartbeatReq) -> HeartbeatResp,
    dispatch_event: DispatchEvent(DispatchEventReq) -> DispatchEventResp,
}

This macro-based system supports:

In-memory transport for testing
HTTP/2 transport for production
Easy addition of new RPC endpoints

The transport layer abstraction is currently defined but not actively used. The primary communication uses the WebSocket-based protocol described above.

Future Enhancements

Sandboxing capabilities through Khronos DAPI
Template shop update broadcasting across workers
Enhanced monitoring and metrics collection
Distributed Mesophyll server deployment

Public API

Internal API

Mesophyll Coordination Layer

Overview

Architecture

Server Component

Client Component

WebSocket Protocol

Connection Establishment

Message Format

Message Types

Server Messages

Hello

DispatchEvent

RunScript

DropWorker

Client Messages

DispatchResponse

Heartbeat

Request/Response Pattern

Database HTTP Endpoints

Tenant State Endpoints

Key-Value Endpoints

Authentication

Request/Response Format

Database Client

Example Usage

Connection Management

Heartbeat System

Automatic Reconnection

Reconnection Handling

Concurrency

Error Handling

Connection Errors

Message Processing

Database Operations

Transport Layer

Future Enhancements

Build docs developers (and LLMs) love

Public API

Internal API

​Overview

​Architecture

​Server Component

​Client Component

​WebSocket Protocol

​Connection Establishment

​Message Format

​Message Types

​Server Messages

​Hello

​DispatchEvent

​RunScript

​DropWorker

​Client Messages

​DispatchResponse

​Heartbeat

​Request/Response Pattern

​Database HTTP Endpoints

​Tenant State Endpoints

​Key-Value Endpoints

​Authentication

​Request/Response Format

​Database Client

​Example Usage

​Connection Management

​Heartbeat System

​Automatic Reconnection

​Reconnection Handling

​Concurrency

​Error Handling

​Connection Errors

​Message Processing

​Database Operations

​Transport Layer

​Future Enhancements

Build docs developers (and LLMs) love

Overview

Architecture

Server Component

Client Component

WebSocket Protocol

Connection Establishment

Message Format

Message Types

Server Messages

Hello

DispatchEvent

RunScript

DropWorker

Client Messages

DispatchResponse

Heartbeat

Request/Response Pattern

Database HTTP Endpoints

Tenant State Endpoints

Key-Value Endpoints

Authentication

Request/Response Format

Database Client

Example Usage

Connection Management

Heartbeat System

Automatic Reconnection

Reconnection Handling

Concurrency

Error Handling

Connection Errors

Message Processing

Database Operations

Transport Layer

Future Enhancements