Gateways and Leaf Nodes

​

Overview

• Gateways: Connect multiple NATS clusters into a super-cluster for geo-distribution
• Leaf nodes: Extend NATS clusters with hub-and-spoke connections for edge computing

​

Gateways

​

Gateway Concept

• Links two named clusters
• Exchanges subscription interest between clusters
• Forwards messages only when interest exists
• Operates at the cluster level, not server level
• Optimizes for wide-area network efficiency

​

Super-Cluster Architecture

• Interest-based routing: Messages only cross gateways when subscribers exist
• Optimistic mode: Initially assumes interest until told otherwise
• Interest-only mode: Explicit interest tracking for efficiency
• Transitioning mode: Automatic mode switching based on traffic patterns

​

Gateway Interest Modes

type GatewayInterestMode byte

const (
    Optimistic     // Send unless no interest
    Transitioning  // Switching to interest-only
    InterestOnly   // Only send with explicit interest
)

• Cluster sends messages across gateway unless explicitly told no interest
• Efficient when most subjects have remote subscribers
• Remote sends “no interest” messages to suppress traffic

• Cluster only sends messages with explicit remote interest
• Efficient when few subjects have remote subscribers
• Requires full interest propagation

• Triggered after too many “no interest” messages (default: 1000)
• Cluster switches to interest-only mode
• Prevents excessive unnecessary traffic

​

Gateway Configuration

gateway {
  name: "us-west"
  listen: "0.0.0.0:7222"
  
  gateways: [
    {
      name: "us-east"
      urls: [
        "nats://gateway1.us-east.example.com:7222"
        "nats://gateway2.us-east.example.com:7222"
      ]
    }
    {
      name: "eu-west"
      urls: [
        "nats://gateway1.eu-west.example.com:7222"
      ]
    }
  ]
}

​

Gateway TLS

gateway {
  name: "us-west"
  listen: "0.0.0.0:7222"
  
  tls {
    cert_file: "./certs/gateway-cert.pem"
    key_file: "./certs/gateway-key.pem"
    ca_file: "./certs/ca.pem"
    verify: true
  }
  
  gateways: [...]
}

​

Gateway Protocol Details

• gatewayCmdGossip: Share gateway topology
• gatewayCmdAllSubsStart: Begin full interest sync
• gatewayCmdAllSubsComplete: Finish interest sync

• Gateway name and URLs
• Cluster information
• Interest mode settings
• Protocol capabilities

​

Reply Subject Handling

• Old format: $GR.<hash>.<subject>
• New format: _GR_.<cluster_hash>.<server_hash>.<subject>

​

Use Cases for Gateways

​

Geographic Distribution

Cluster US-WEST (3 servers) <--gateway--> Cluster US-EAST (3 servers)
                                                      ^
                                                      |
                                                   gateway
                                                      |
                                                      v
                                          Cluster EU-WEST (3 servers)

• Each region has a local cluster (low latency)
• Gateways connect regions (high latency tolerant)
• Clients connect to local cluster
• Messages reach subscribers in any region

​

Failover Between Regions

1. Client connects to local cluster
2. Local cluster fails
3. Client reconnects to remote cluster via gateway
4. Application continues without message loss

​

Data Sovereignty

• Use accounts to isolate sensitive data
• Configure imports/exports to control cross-region flow
• Messages only cross gateways when explicitly allowed

​

Cost Optimization

• Interest-only mode minimizes unnecessary traffic
• Messages only traverse expensive WAN links when needed
• Local clusters handle regional traffic locally

​

Leaf Nodes

​

Leaf Node Concept

• Connects a server (spoke) to a cluster (hub)
• Appears as a single client to the hub
• Supports bidirectional messaging
• Automatically reconnects to hub cluster
• Can operate in isolated mode

​

Hub and Spoke vs Spoke

• Hub: The server/cluster accepting leaf connections
• Spoke: The server initiating the leaf connection
• Solicited: Spoke-initiated outbound connection
• Unsolicited: Hub-accepted inbound connection

​

Leaf Node Architecture

     Hub Cluster (3 servers)
       /       |        \
    Edge1    Edge2    Edge3

  Corporate Cluster
         |
    Department Server
         |
      Employees

  Production Cluster
         |
    Developer Laptop

​

Leaf Node Configuration

​

Hub Configuration

leafnodes {
  listen: "0.0.0.0:7422"
  
  authorization {
    users: [
      {user: "leaf1", password: "password1", account: "ACCOUNT1"}
      {user: "leaf2", password: "password2", account: "ACCOUNT2"}
    ]
  }
}

​

Spoke Configuration

leafnodes {
  remotes: [
    {
      url: "nats-leaf://leaf1:[email protected]:7422"
      account: "LOCAL_ACCOUNT"
    }
  ]
}

​

Leaf Node Features

​

Account Binding

leafnodes {
  remotes: [
    {
      url: "nats-leaf://hub.example.com:7422"
      account: "EDGE_USERS"      # Local account
      credentials: "./leaf.creds" # Maps to hub account
    }
  ]
}

​

TLS for Leaf Nodes

leafnodes {
  remotes: [
    {
      url: "tls://hub.example.com:7422"
      tls {
        cert_file: "./certs/leaf-cert.pem"
        key_file: "./certs/leaf-key.pem"
        ca_file: "./certs/ca.pem"
      }
    }
  ]
}

​

WebSocket Leaf Nodes

leafnodes {
  remotes: [
    {
      url: "wss://hub.example.com/leafnode"
    }
  ]
}

​

Compression

leafnodes {
  remotes: [
    {
      url: "nats://hub.example.com:7422"
      compression: "s2_auto"
    }
  ]
}

​

Leaf Node Isolation

• Don’t receive subscriptions from other leaf nodes
• Only see traffic from hub cluster
• Prevents east-west traffic between leaf nodes
• Useful for security and performance

leafnodes {
  remotes: [
    {
      url: "nats://hub.example.com:7422"
      isolated: true
    }
  ]
}

​

Loop Detection

• Loop detection subject: $LDS.<random>
• If leaf receives its own loop detection message, loop detected
• Connection closed with 30-second reconnect delay
• Prevents cluster connecting to itself via leaf

​

Leaf Node Reconnection

• Multiple hub URLs supported for failover
• Automatic reconnection on disconnect
• Subscription state restored on reconnect
• From server/leafnode.go:50-57, special delays for error conditions:
  • Loop detected: 30 seconds
  • Permission violation: 30 seconds
  • Same cluster name: 30 seconds

​

Multi-Region Connectivity Patterns

​

Hybrid: Clusters + Gateways + Leaf Nodes

Region US:
  Cluster US-WEST (3 servers) <--gateway--> Cluster US-EAST (3 servers)
       ^
       | (leaf)
       |
    Edge Servers (leaf nodes)

• Regional clusters for low latency
• Gateways for cross-region
• Leaf nodes for edge locations

​

Active-Active Multi-Region

Cluster A <--gateway--> Cluster B <--gateway--> Cluster C
   ^                        ^                        ^
   |                        |                        |
 Clients                 Clients                 Clients

​

Hub with Regional Spokes

         Hub Cluster
         /    |    \
      Leaf  Leaf  Leaf
     EMEA   APAC   AMER

​

Configuration Examples from Source

​

Gateway with Multiple Remotes

gateway {
  name: "production-us"
  listen: "0.0.0.0:7222"
  
  # Advertise specific URLs
  advertise: "gateway.prod-us.example.com:7222"
  
  # Connect to other clusters
  gateways: [
    {name: "production-eu", urls: ["nats://gw1.prod-eu.example.com:7222"]}
    {name: "production-ap", urls: ["nats://gw1.prod-ap.example.com:7222"]}
  ]
  
  # Don't accept unknown gateways
  reject_unknown: true
}

​

Leaf Node with Credentials

leafnodes {
  remotes: [
    {
      urls: [
        "nats://hub1.example.com:7422"
        "nats://hub2.example.com:7422"
        "nats://hub3.example.com:7422"
      ]
      credentials: "/etc/nats/leaf.creds"
      account: "EDGE"
    }
  ]
}

​

Performance Considerations

​

Gateway Performance

• Interest propagation overhead: Interest-only mode trades setup cost for reduced traffic
• RTT sensitivity: High RTT impacts mode transition decisions
• Connection pooling: From server/gateway.go:140, gateways maintain ordered connections
• Ping interval: Max 15 seconds from server/gateway.go:58

​

Leaf Node Performance

• Hub cluster impact: Each leaf appears as one client
• Subject interest: All leaf subscriptions propagated to hub
• Message amplification: Hub messages reach all leaf subscribers
• Isolation benefit: Isolated leafs reduce cross-leaf traffic

​

Best Practices

​

Gateway Deployment

1. Name gateways consistently: Use geographic or functional names
2. Configure all connections: Each cluster should list all other clusters
3. Use TLS: Encrypt cross-region traffic
4. Monitor interest modes: Track mode transitions
5. Set reject_unknown: Prevent unauthorized gateway connections

​

Leaf Node Deployment

1. Use credentials: JWT-based authentication preferred
2. Multiple hub URLs: Configure failover URLs
3. Account mapping: Plan account topology carefully
4. Consider isolation: Use for edge security
5. Compression for WAN: Enable for bandwidth-limited connections

​

Capacity Planning

1. Gateway bandwidth: Monitor cross-region traffic
2. Hub capacity: Each leaf consumes hub resources
3. Interest churn: Frequent subscriptions trigger interest updates
4. Subject cardinality: Many unique subjects increase interest state

​

Troubleshooting

​

Gateway Issues

• Connection state per gateway
• Interest mode per account
• Message and byte counts
• Number of “no interest” suppressions

​

Leaf Node Issues

• Leaf connection status
• Account bindings
• Subscription counts
• RTT and compression stats

• Loop detection: Check cluster names don’t match
• Permission errors: Verify account permissions
• Connection failures: Check network and authentication

​

Next Steps