Edge network

The edge network serves as the entry point to Ably’s realtime infrastructure and is core to the platform’s ability to deliver reliable, low-latency messaging services globally.

​

Why the edge network matters

The edge network must ensure that clients can reliably connect to the closest available datacenter, providing optimal performance whilst maintaining resilience against various types of failures. Given that Ably is mission-critical for many applications, the reliability of the edge network is essential; any connectivity issues can potentially impact thousands or millions of end users.

​

Edge network architecture

Ably’s edge network is designed to provide reliable, low-latency connectivity to the Ably platform from anywhere in the world. The architecture leverages multiple AWS services and custom infrastructure components to ensure resilience, performance, and security.

​

Global deployment across multiple regions

Ably operates in multiple regions around the world, with servers distributed across more than 15 physical datacenters within the AWS network. This global distribution ensures that clients can connect to a nearby datacenter, minimizing the latency between the client and the entry point to Ably’s network. Key benefits:

• Each datacenter operates independently with its own dedicated infrastructure
• Issues affecting one datacenter do not propagate to others
• Service availability is maintained even during regional outages
• Data residency requirements can be met by keeping data within specific geographic regions

​

AWS CloudFront and network load balancers

Client connections to Ably are handled through a combination of AWS CloudFront for global edge distribution and AWS EC2 Network Load Balancers (NLBs) for traffic routing within each region.

​

AWS CloudFront

AWS CloudFront serves as the primary entry point to Ably’s network, with over 700 edge locations globally:

• When a client attempts to connect to Ably, the request is first routed to the nearest CloudFront edge location
• This reduces the public internet transit time
• Clients connect to a nearby edge node rather than traversing the entire distance to an Ably datacenter

​

Network Load Balancers

Behind CloudFront, each Ably region employs AWS Network Load Balancers:

• Operate at the transport layer
• Handle millions of requests per second
• Maintain ultra-low latencies
• Distribute traffic to frontend servers responsible for establishing and maintaining client connections

This architecture allows Ably to efficiently scale the number of concurrent connections by adding more frontend servers as needed.

​

DNS organization and latency-based routing

Ably uses DNS-based latency routing to direct clients to the nearest available datacenter. Primary endpoint: main.realtime.ably.net When a client performs a DNS lookup for this endpoint, the DNS service resolves to the closest datacenter to the client’s location. This latency-based routing ensures that clients connect to the datacenter with the lowest network latency, maximizing the responsiveness of the service. Key features:

• TTL of 60 seconds: Allows for relatively quick rerouting of traffic if a datacenter becomes unhealthy
• Continuous health monitoring: Datacenters are continuously monitored
• Automatic traffic redirection: DNS routing can be modified within minutes to direct traffic away from affected datacenters
• Resilient DNS infrastructure: Alternate DNS provider is used for fallback endpoints to eliminate single points of failure

​

Fallback endpoints and secondary domains

While DNS-based routing is effective for normal operations, it has limitations in certain failure scenarios. To address these limitations, Ably implements a fallback mechanism in all client libraries.

​

Fallback mechanism

If a client cannot connect to the primary endpoint, it automatically attempts to connect using alternative endpoints:

• Direct connections to specific datacenters, bypassing default latency-based DNS routing
• Fallback endpoints cycle through up to 6 globally distributed locations
• Progressive connection timeout strategies ensure rapid recovery from transient issues

​

Secondary domain

Ably operates a completely segregated secondary domain, ably-realtime.com:

• Designed to cater to any DNS failures on the primary ably.net domain
• Uses a different DNS provider from the primary domain
• Ensures that domain-level issues do not affect both domains simultaneously

​

Protocol-level resilience

Ably’s client libraries implement intelligent retry logic to handle temporary connectivity issues:

• Adaptive retry strategy: Uses shorter intervals for transient issues and longer intervals for persistent problems
• Connection recovery: When a connection is re-established after a disconnection, client libraries attempt to recover the previous connection state, including message continuity
• Automatic reconnection: Clients automatically attempt to reconnect via alternative endpoints if a connection attempt fails
• Connection quality monitoring: Continuous monitoring triggers proactive reconnection when performance degrades

These client-side mechanisms work in conjunction with the server-side infrastructure to provide reliable connectivity even during significant infrastructure disruptions.

​

Infrastructure redundancy

Beyond DNS and client-side fallbacks, Ably’s infrastructure includes multiple layers of redundancy:

​

Protection against denial of service attacks

As a critical infrastructure provider for many businesses, Ably is a potential target for denial of service (DoS) attacks. The edge network includes multiple layers of protection to detect, mitigate, and absorb such attacks without impacting legitimate traffic.

​

Layer 3/4 DoS protection with AWS Shield Advanced

Ably utilizes AWS Shield Advanced to protect against volumetric DDoS attacks at the network and transport layers (Layers 3 and 4). These attacks typically involve flooding the target with a high volume of packets or connection attempts. Protection features:

• Automatic attack detection: Continuous monitoring of traffic patterns to detect anomalies
• Automatic mitigation: Filters malicious traffic while allowing legitimate traffic to pass through
• Global network absorption: Leverages AWS’s global network infrastructure to absorb and diffuse attack traffic
• Expert support: Access to the AWS Shield Response Team (SRT) during attacks

These protections ensure that even large-scale volumetric attacks can be effectively mitigated before reaching Ably’s infrastructure.

​

Layer 7 protection with CloudFront WAF

While Shield Advanced is effective against volumetric attacks, more sophisticated attacks target the application layer (Layer 7) by exploiting specific application behaviors or vulnerabilities. Ably uses the CloudFront Web Application Firewall (WAF) to protect against these application-layer attacks. WAF capabilities:

• Custom rules: Identify and block malicious requests based on their characteristics
• Rate limiting: Limit the rate of requests from specific IP addresses or address ranges
• Geographic blocking: Temporarily block traffic from specific regions if attacks originate there
• Bot management: Distinguish between legitimate bots (such as search engine crawlers) and malicious bots

WAF works in conjunction with other AWS security services, allowing for coordinated defense across multiple layers of the infrastructure.

​

Scalable infrastructure for absorbing traffic spikes

In addition to filtering malicious traffic, Ably’s infrastructure is designed to scale rapidly in response to traffic spikes:

• Automatic scaling: Frontend services automatically scale to handle increased connection rates
• Connection management: Prevents a single client or group of clients from consuming excessive resources
• Multi-level rate limiting: Implemented per account, per application, per key, per token, and per IP address
• Attack detection: Mechanisms to detect and mitigate connection-based attacks (such as slow loris attacks or connection flooding)
• Graceful load shedding: Under extreme load, temporarily reject new connections or non-critical requests to preserve core functionality

This scalable infrastructure ensures that even if attack traffic reaches Ably’s application servers, the impact on legitimate users is minimized.

​

Global connectivity considerations

Operating a truly global edge network presents unique challenges and considerations that Ably has addressed in its architecture.

​

Geographic coverage and edge presence

Ably’s edge network is designed to provide low-latency connectivity from virtually anywhere in the world:

• Combination of AWS CloudFront’s extensive network and Ably’s strategically located datacenters
• Most clients can connect to an entry point with minimal network latency
• Geographic distribution is continuously reviewed and optimized based on traffic patterns
• Additional datacenters can be added as the user base expands into new regions

​

China connectivity

Ably’s service works in China, with customers successfully using the platform to serve users globally, including within China:

• Global network with over 700 edge locations provides connectivity to Chinese users
• China operates a national firewall that can block access to foreign websites and services without notice
• While Ably is not currently aware of any blocks affecting its services, potential firewall changes could impact service availability
• Ably has implemented specific strategies to ensure reliable service in China

​

Protocol support and transport optimization

The edge network supports a range of transport protocols to ensure connectivity across diverse client environments:

• WebSockets: Preferred transport for realtime connections
• Comet (long-polling HTTP): For clients or network settings where WebSockets are not available or are blocked
• Automatic transport selection: Handled by Ably’s client libraries based on the client’s environment and network conditions

This ensures that clients can connect to Ably even in restrictive network environments, such as corporate networks with restrictive firewalls or proxy servers.

​

Handling network transitions and interruptions

Mobile clients frequently experience network transitions as they move between different connectivity types (WiFi to cellular, for example) or through areas with varying signal strength:

• When a client transitions between networks, it will lose its existing connections
• The edge network’s design allows clients to quickly reestablish connections and resume their session state
• This minimizes the impact of network transitions on the application experience

​

Firewall and network configuration

If you need to configure firewalls or network security devices to allow connections to Ably, the following information will help you set up the necessary rules.

​

Ports

Ably’s client libraries use the following ports:

• Port 443 - HTTPS traffic over TLS (primary for all WebSocket and HTTP connections)
• Port 80 - HTTP traffic (only when TLS is explicitly disabled, not recommended)

Protocol adapters and integrations use additional ports:

• Port 5671 - Ably queues over AMQP (TLS only)
• Port 61614 - Ably queues over STOMP (TLS only)
• Port 8883 - MQTT adapter over TLS
• Port 1883 - MQTT adapter unencrypted (not recommended)

​

Domains and endpoints

Core Ably endpoints:

• rest.ably.io - REST API requests
• realtime.ably.io - Realtime WebSocket connections
• a.ably-realtime.com through e.ably-realtime.com - Fallback hosts for reliability

Additional connectivity endpoints:

• internet-up.ably-realtime.com - General connectivity checks
• ws-up.ably-realtime.com - WebSocket connectivity checks (ably-js v2)

Protocol adapter endpoints:

• mqtt.ably.io - MQTT adapter
• pubnub-rest.ably.io - PubNub adapter
• pusher-rest.ably.io and pusher-realtime.ably.io - Pusher adapter
• us-east-1-a-queue.ably.io - Ably Queues (US East 1)
• eu-west-1-a-queue.ably.io - Ably Queues (EU West 1)

​

DNS CNAME records

Ably’s default endpoints use DNS CNAME records with the following targets:

• rest.ably.io and realtime.ably.io → main.realtime.ably.net
• a.ably-realtime.com → main.a.fallback.ably-realtime.com
• b.ably-realtime.com → main.b.fallback.ably-realtime.com
• c.ably-realtime.com → main.c.fallback.ably-realtime.com
• d.ably-realtime.com → main.d.fallback.ably-realtime.com
• e.ably-realtime.com → main.e.fallback.ably-realtime.com

​

IP addresses

​

Enterprise customers

​

Next steps