​

Cloudflare Global Outage

​

Overview

​

Root Cause Analysis

​

Primary Technical Cause

​

The Technical Details

// Simplified illustration
fn load_bot_features(file: &str) -> Result<BotFeatures, ConfigError> {
    let features = parse_config(file)?;
    
    // Hard-coded assumption: features.len() <= 200
    if features.len() > 200 {
        return Err(ConfigError::TooManyFeatures);
    }
    
    Ok(features)
}

// Somewhere in the proxy code:
let bot_features = load_bot_features(config_file).unwrap();  // PANIC!

​

Why It Cascaded Globally

1. Over-coupled components: Bot-management → proxy → Workers → Dashboard formed a single failure domain
2. Automated propagation: Bad config spread before detection
3. Internal tool dependency: Engineers couldn’t access Dashboard (needed Turnstile, which was behind failing proxies)
4. No graceful degradation: Proxy panic caused complete failure, not reduced functionality

​

Timeline

​

Impact Assessment

​

Global Internet Impact

​

High-Profile Services Affected

​

Cloudflare Internal Impact

​

Contributing Factors

​

Hard-Coded Feature Limit

// Instead of unwrap() which panics:
let bot_features = load_bot_features(config_file).unwrap();

// Use graceful error handling:
let bot_features = match load_bot_features(config_file) {
    Ok(features) => features,
    Err(e) => {
        log::error!("Failed to load bot features: {}", e);
        // Use cached version or safe defaults
        get_cached_bot_features().unwrap_or_default()
    }
};

​

Lack of Configuration Validation

• Size limits: No monitoring of file growth
• Duplicate detection: Duplicate entries not flagged
• Schema validation: No enforcement of expected structure
• Checksum verification: No integrity checks before deployment
• Pre-deployment testing: No staging environment validation

​

Automated, Timed Propagation

# Simplified illustration of problematic approach
while True:
    config = generate_bot_config()  # No validation!
    push_to_all_nodes(config)       # No canary!
    time.sleep(300)                 # 5 minutes

# Better approach with validation and canary
def deploy_config(config):
    # Validate before deployment
    if not validate_config(config):
        alert("Config validation failed")
        return False
    
    # Canary deployment
    canary_nodes = get_canary_fleet(percentage=5)
    deploy_to_nodes(canary_nodes, config)
    
    # Monitor canary
    if not monitor_health(canary_nodes, duration=600):
        rollback(canary_nodes)
        alert("Canary failed, rollback triggered")
        return False
    
    # Gradual rollout
    for percentage in [10, 25, 50, 100]:
        nodes = get_fleet(percentage)
        deploy_to_nodes(nodes, config)
        if not monitor_health(nodes, duration=300):
            rollback_all()
            return False
    
    return True

​

Over-Coupled System Components

• Provide alternate access path to Dashboard (direct origin, VPN)
• Don’t put internal tooling behind the same infrastructure as customer traffic
• Implement circuit breakers to prevent cascading failures
• Design for graceful degradation (disable bot-management if config fails)

​

Internal Change Misclassified as Low-Risk

• Classified as low-risk routine maintenance
• Not subject to full change control process
• Not tested in staging environment
• Approved quickly without deep review

​

Mitigation & Recovery Actions

​

Cloudflare’s Response

​

Residual Behavior

• Latency fluctuations persisted briefly as caches warmed up
• Dependent services recovered at different speeds based on node propagation order
• Some DNS caches held stale failure information temporarily
• Customer applications needed to retry failed requests

​

Lessons Learned

​

1. Treat All Config Surfaces as High-Risk

class BotFeatureConfig:
    MAX_FEATURES = 250  # Allow headroom above expected 200
    MAX_FILE_SIZE = 10 * 1024 * 1024  # 10 MB
    
    def generate(self) -> Optional[str]:
        # Query database
        features = self.query_bot_features()
        
        # Validate: no duplicates
        if len(features) != len(set(f['id'] for f in features)):
            log.error("Duplicate features detected!")
            return None
        
        # Validate: within limits
        if len(features) > self.MAX_FEATURES:
            log.error(f"Too many features: {len(features)}")
            return None
        
        # Generate config file
        config = self.serialize(features)
        
        # Validate: size check
        if len(config) > self.MAX_FILE_SIZE:
            log.error(f"Config too large: {len(config)} bytes")
            return None
        
        # Generate checksum
        checksum = hashlib.sha256(config.encode()).hexdigest()
        log.info(f"Generated config: {len(features)} features, checksum {checksum}")
        
        return config

​

2. Fallback Paths Prevent Catastrophic Failure

fn serve_request(req: Request) -> Response {
    // Try to apply bot management
    let bot_decision = match apply_bot_management(&req) {
        Ok(decision) => decision,
        Err(e) => {
            // Log error but don't fail the request
            log::warn!("Bot management failed: {}, allowing request", e);
            BotDecision::Allow  // Safe default
        }
    };
    
    if bot_decision == BotDecision::Block {
        return Response::forbidden();
    }
    
    // Continue serving the request
    proxy_to_origin(&req)
}

​

3. Propagation Must Be Controlled and Observable

​

4. Monitor Assumptions—Not Just Metrics

if current_size > baseline_size * 1.5:
    alert("Config file size anomaly")

if len(features) != len(set(features)):
    alert("Duplicate features detected")

if restart_count > 3 in last_5_minutes:
    alert("Proxy restart loop detected")

affected_nodes = count_failing_nodes()
if affected_nodes > total_nodes * 0.1:
    alert("Widespread failure detected")

​

5. Internal Changes Need Staging and Risk Assessment

​

6. Access Paths Must Survive Outages

​

Key Takeaways

​

References

• Official Cloudflare Post-Mortem