​

Failover Procedures

​

Failover Overview

1. Detects the failure through health checks or error tracking
2. Selects a new primary from healthy secondary nodes
3. Promotes the secondary to serve as the new primary
4. Updates routing to direct traffic to the new primary
5. Schedules replication to restore data to other secondaries

​

Failover Strategies

​

Per-Repository Elector (Recommended)

[failover]
enabled = true
election_strategy = "per_repository"

• Best availability: one node failure doesn’t affect repositories on other nodes
• Better load distribution across the cluster
• Granular failover: only affected repositories fail over
• Fastest recovery time

• PostgreSQL database for storing primary assignments
• All Praefect nodes must have database access

​

SQL Elector

[failover]
enabled = true
election_strategy = "sql"

1. Each Praefect stores its view of node health in the database
2. When a primary fails, Praefects vote on a replacement
3. The candidate with the least replication lag wins
4. A majority of Praefects must agree on the new primary

​

Local Elector (Development Only)

[failover]
enabled = true
election_strategy = "local"

• No database required
• No coordination between Praefect instances
• Useful for local development environments
• Not suitable for production: different Praefects may disagree on primary

​

Failover Disabled

[failover]
enabled = false

• Virtual storage becomes unavailable if the configured primary fails
• Manual intervention required to restore service
• Useful for debugging or performing controlled maintenance

​

Health Monitoring

​

Error Tracking

[failover]
enabled = true
error_threshold_window = "1m"
read_error_threshold_count = 10
write_error_threshold_count = 5

• Track errors per node within a time window
• Mark a node unhealthy if errors exceed the threshold
• Read errors are weighted separately from write errors
• Unhealthy nodes are excluded from primary election

​

Health Checks

[failover]
bootstrap_interval = "1s"    # Initial check interval at startup
monitor_interval = "3s"      # Regular check interval during operation

• Gitaly node is reachable
• gRPC connection is healthy
• Node can respond to basic RPCs

​

Primary Selection

​

Eligibility Requirements

1. Health: Node must be currently healthy
2. Connectivity: Node must be reachable by majority of Praefects
3. Data consistency: Node must not be significantly behind

​

Selection Criteria

1. Replication lag: Prefer nodes with fewer pending replication jobs
2. Data recency: Choose the node with the most up-to-date data
3. Position: If all else equal, select based on configuration order

​

Read-Only Mode

​

Why Read-Only?

​

Recovery from Read-Only

1. All missing data is replicated from another node, or
2. Administrator accepts data loss using accept-dataloss

# Praefect's reconciler automatically schedules replication
# No manual intervention required if a complete copy exists

# When complete data is unrecoverable, designate authoritative version
praefect -config /path/to/config.toml accept-dataloss \
  -virtual-storage default \
  -relative-path @hashed/ab/cd/abcd1234.git \
  -authoritative-storage gitaly-2

​

Identifying Data Loss

# Check all virtual storages
praefect -config /path/to/config.toml dataloss

# Check specific virtual storage
praefect -config /path/to/config.toml dataloss -virtual-storage default

Virtual storage: default
  Repositories:
    @hashed/ab/cd/abcd1234.git:
      Primary: gitaly-2 (1 behind)
      Outdated secondaries:
        gitaly-1: 1 change(s) behind, last healthy at 2021-01-15 10:30:00

• Repository’s current primary is gitaly-2
• gitaly-1 is missing 1 change from when it was primary
• gitaly-1 was last seen healthy at the shown timestamp

​

Failover Testing

​

Planned Failover Test

1. Verify current state:
   # Check which node is primary
   praefect -config /path/to/config.toml dataloss
   
2. Stop a Gitaly node:
   systemctl stop gitaly@gitaly-1
   
3. Monitor Praefect logs:
   tail -f /var/log/gitlab/praefect/current | grep -i failover
   
4. Verify failover occurred:
   praefect -config /path/to/config.toml dataloss
   
5. Test repository access:
   git clone [email protected]:group/project.git
   
6. Restart the node:
   systemctl start gitaly@gitaly-1
   
7. Verify replication:
   # Check replication queue clears
   curl -s http://praefect:10101/metrics | grep replication_queue_depth
   

​

Unplanned Failure Simulation

# Simulate network partition
iptables -A INPUT -s <gitaly-node-ip> -j DROP
iptables -A OUTPUT -d <gitaly-node-ip> -j DROP

# Wait for failover (monitor logs)
# Then restore connectivity
iptables -D INPUT -s <gitaly-node-ip> -j DROP
iptables -D OUTPUT -d <gitaly-node-ip> -j DROP

​

Monitoring Failover Events

​

Key Log Messages

level=info msg="primary node changed" virtual_storage=default repository=@hashed/... old_primary=gitaly-1 new_primary=gitaly-2
level=warn msg="node marked unhealthy" virtual_storage=default node=gitaly-1 reason="health check failed"
level=info msg="node marked healthy" virtual_storage=default node=gitaly-1

​

Key Metrics

# Node health status (1=healthy, 0=unhealthy)
gitaly_praefect_node_health{virtual_storage="default",node="gitaly-1"}

# Number of repositories in read-only mode
gitaly_praefect_read_only_repositories{virtual_storage="default"}

# Primary node changes
rate(gitaly_praefect_primary_elections_total[5m])

​

Best Practices

1. Use per-repository election: Provides best availability and granularity
2. Configure error thresholds: Catch soft failures before they impact users
3. Monitor replication lag: Large lag increases data loss risk during failover
4. Test regularly: Verify failover works before you need it
5. Plan for data loss: Document procedures for handling accept-dataloss scenarios
6. Size appropriately: Ensure remaining nodes can handle load after failover

​

Recovering Failed Nodes

1. Praefect detects recovery through health checks
2. Node marked healthy and eligible for primary election
3. Replication scheduled to bring node up-to-date
4. Node rejoins cluster once replication completes

curl -s http://praefect:10101/metrics | grep 'replication_queue.*gitaly-1'

​

Next Steps