Drift Prevention - Safe Settings

Configuration drift occurs when repository settings are modified outside of Safe Settings, causing actual configurations to diverge from your policy-as-code definitions. Safe Settings provides multiple mechanisms to prevent and correct drift.

What is Configuration Drift?

Drift happens when:

Users manually change settings through GitHub UI
Automation tools modify configurations
Branch protections are deleted or weakened
Team permissions are altered
Repository settings are changed

Without drift prevention, these manual changes can:

Create security vulnerabilities
Violate compliance policies
Cause inconsistencies across repositories
Undermine policy-as-code governance

Drift Prevention Strategies

Safe Settings uses two complementary approaches:

Webhook-based detection - Immediate response to changes
Scheduled synchronization - Periodic convergence

Webhook-Based Drift Prevention

Safe Settings listens to GitHub webhook events and automatically reverts unauthorized changes.

Monitored Events

Safe Settings responds to these webhook events to prevent drift:

Branch Protection Changes

Event: branch_protection_rule
Triggers: created, edited, deleted

Example scenario:

User disables “Require pull request reviews” in GitHub UI
Webhook fires: branch_protection_rule.edited
Safe Settings detects change doesn’t match configuration
Automatically restores required reviews setting

Repository Settings Changes

Event: repository.edited
Triggers: renamed, visibility changed, topics modified, default branch changed

Example scenario:

User changes repository from private to public
Webhook fires: repository.edited
Safe Settings compares against settings configuration
Reverts repository to private if configured

Team Permission Changes

Events:
  - member.added
  - member.removed
  - team.added_to_repository
  - team.removed_from_repository
  - team.edited

Example scenario:

User manually grants team admin access to repository
Webhook fires: team.added_to_repository
Safe Settings checks configured permissions
Adjusts to configured permission level (e.g., write)

Ruleset Changes

Event: repository_ruleset
Triggers: created, edited, deleted

Example scenario:

User modifies ruleset to remove status check requirements
Webhook fires: repository_ruleset.edited
Safe Settings restores status check requirements

Custom Property Changes

Event: custom_property_values
Triggers: set, updated

Example scenario:

User sets custom property environment=production
Webhook fires: custom_property_values
Safe Settings applies suborg configuration for production repos

How Webhook Drift Prevention Works

Configuration for Webhooks

Webhook-based drift prevention is automatic. Ensure your GitHub App has these permissions:

Repository administration: Read & write
Repository hooks: Read & write
Organization administration: Read & write

Webhook Event Examples

Example 1: Branch Protection Drift

branches:
  - name: main
    protection:
      required_pull_request_reviews:
        required_approving_review_count: 2
        dismiss_stale_reviews: true
      enforce_admins: true

Example 2: Team Permission Drift

teams:
  - name: developers
    permission: write
  - name: security-team
    permission: admin

Scheduled Synchronization

Scheduled sync provides a safety net by periodically converging all repositories to their configured state, catching any drift that webhooks might miss.

Why Scheduled Sync?

Webhooks aren’t always guaranteed to be delivered:

Network failures
Webhook delivery failures
Service outages
GitHub webhook limits
Missed events during maintenance

Scheduled sync ensures eventual consistency.

Configuring Scheduled Sync

Set the CRON environment variable to run Safe Settings on a schedule.

# Run at the start of every hour
CRON=0 * * * *

Cron Syntax

Safe Settings uses node-cron syntax:

┌────────────── second (optional, 0-59)
│ ┌──────────── minute (0-59)
│ │ ┌────────── hour (0-23)
│ │ │ ┌──────── day of month (1-31)
│ │ │ │ ┌────── month (1-12)
│ │ │ │ │ ┌──── day of week (0-7, 0 and 7 are Sunday)
│ │ │ │ │ │
│ │ │ │ │ │
* * * * * *

Common Cron Patterns

Pattern	Description	Use Case
`/5 * * *`	Every 5 minutes	Testing, high-security
`0 * * * *`	Every hour	Standard production
`0 /2 * *`	Every 2 hours	Balanced approach
`0 /6 * *`	Every 6 hours	Light sync
`0 0 * * *`	Daily at midnight	Low-frequency
`0 2 * * *`	Daily at 2 AM	Off-peak hours
`0 0 * * 0`	Weekly on Sunday	Minimal sync
`0 9 * * 1-5`	Weekdays at 9 AM	Business hours

How Scheduled Sync Works

Scheduled Sync Behavior

During scheduled sync:

Load all configurations: Org, suborg, and repo settings
Process each repository: Apply configuration hierarchy
Detect differences: Compare current GitHub state with expected state
Apply only changes: Only make API calls when drift is detected
Create check runs: Document sync results

Performance Considerations

Scheduled sync is optimized for organizations with thousands of repositories:

Only loads changed configuration files (not all files every time)
Only makes API calls when drift is detected
Respects rate limits automatically (via Probot)
Completes within GitHub App token lifetime (1 hour)

See Performance Optimization for details.

Comparison Logic

Both webhook and scheduled sync use intelligent comparison to detect drift.

What Gets Compared

Safe Settings compares:

Repository settings (visibility, features, default branch)
Branch protection rules
Required status checks
Team permissions
Collaborator permissions
Labels
Milestones
Autolinks
Environments
Variables
Rulesets
Custom properties

How Comparison Works

The compareDeep function generates detailed differences:

{
  "branches": [{
    "name": "main",
    "protection": {
      "required_pull_request_reviews": {
        "required_approving_review_count": 2,
        "dismiss_stale_reviews": true
      },
      "enforce_admins": true
    }
  }]
}

Smart Comparison Features

Ignores irrelevant fields:

// Ignored during comparison:
- url fields (resource URLs)
- timestamps
- GitHub-generated IDs

Handles different data structures:

// Properly compares:
- Arrays with different ordering
- Nested objects
- Case-sensitive vs case-insensitive fields

Only updates when necessary:

if (results.hasChanges) {
  // Make API call to update
} else {
  // Skip - no actual changes
}

Monitoring Drift

Check Runs

Every drift detection and correction creates a check run in your admin repository:

✅ safe-settings

Sync completed for 45 repositories

Drift detected and corrected:
  - repo-api: Branch protection (required_approving_review_count)
  - repo-web: Team permissions (developers: admin → write)
  - repo-service: Repository settings (visibility: public → private)

No changes needed: 42 repositories

Check Run Details

Click into check runs to see detailed changes:

Repository: repo-api

Modifications:
{
  "branches": [{
    "name": "main",
    "protection": {
      "required_pull_request_reviews": {
        "required_approving_review_count": 2  // Changed from 1
      }
    }
  }]
}

API Calls Made:
  PUT /repos/org/repo-api/branches/main/protection

Logging

Safe Settings logs all drift detection:

# View drift detection logs
docker logs safe-settings | grep "drift\|hasChanges"

# Example output:
[2024-03-15T10:30:45Z] INFO: Drift detected in repo-api
[2024-03-15T10:30:45Z] INFO: hasChanges: true
[2024-03-15T10:30:46Z] INFO: Applied corrections to repo-api

Best Practices

Choose Appropriate Sync Frequency

High-security environments:

# Sync every 30 minutes
CRON=*/30 * * * *

Standard production:

# Sync every 2-6 hours
CRON=0 */2 * * *

Low-risk environments:

# Sync daily
CRON=0 2 * * *

Consider API Rate Limits

More frequent syncs consume more API quota:

Webhooks: Minimal API usage (only affected repos)
Scheduled sync: API calls for all repos (even without changes)

Balance security needs with API limits:

# For 1000 repos:
# Hourly sync = ~1000 API calls/hour
# Every 6 hours = ~167 API calls/hour
# Daily = ~42 API calls/hour

Schedule During Off-Peak Hours

Run heavy syncs when developers are less active:

# Daily at 2 AM
CRON=0 2 * * *

# Twice daily: 2 AM and 2 PM
CRON=0 2,14 * * *

Combine Both Approaches

Use webhooks for immediate response + scheduled sync as backup:

# Webhooks handle most drift immediately
# Scheduled sync catches anything webhooks missed
CRON=0 */6 * * *  # Every 6 hours

Monitor Drift Patterns

Review check runs to identify:

Frequently drifting repositories
Common types of drift
Users making unauthorized changes
Gaps in your policies

Use insights to:

Educate teams on policy
Adjust configurations
Identify process gaps

Handle Legitimate Changes

If drift corrections are unwanted: Option 1: Update configuration to match desired state

# Update settings.yml to reflect new requirements
branches:
  - name: main
    protection:
      required_approving_review_count: 1  # Reduced from 2

Option 2: Use external management for specific settings

# Allow manual management of status checks
required_status_checks:
  contexts:
    - "{{EXTERNALLY_DEFINED}}"

Option 3: Exclude specific repositories

deployment-settings.yml

restrictedRepos:
  exclude:
    - special-repo  # Manual management

Troubleshooting

Drift Not Being Corrected

Problem: Manual changes aren’t being reverted. Possible causes:

Webhooks not configured

# Check GitHub App webhook configuration
# Verify webhook events are enabled

Repository is restricted

# Check deployment-settings.yml
restrictedRepos:
  exclude:
    - problem-repo  # This repo is excluded!

Setting matches configuration

# Manual change actually matches your config
# Update your configuration if this is drift

Scheduled Sync Not Running

Problem: Cron job doesn’t seem to execute. Debug steps:

Verify CRON syntax

# Test on crontab.guru
# Ensure format is correct

Check environment variable

docker exec safe-settings env | grep CRON

Review application logs

docker logs safe-settings | grep -i cron

Verify app is running
```
docker ps | grep safe-settings
```

Too Many API Calls

Problem: Hitting GitHub API rate limits. Solutions:

Reduce sync frequency

# Change from every hour to every 6 hours
CRON=0 */6 * * *

Use restricted repos

# Manage fewer repositories
restrictedRepos:
  include:
    - critical-*

Monitor API usage

# Check rate limit status
gh api rate_limit

Unexpected Drift Corrections

Problem: Safe Settings is reverting legitimate changes. Cause: Configuration doesn’t reflect desired state. Solution: Update configuration to match requirements:

# If Safe Settings keeps reverting a change,
# update your configuration to allow it:

branches:
  - name: main
    protection:
      # Safe Settings was reverting this to true
      # Update config to match desired state:
      enforce_admins: false

Advanced Configuration

Repository-Specific Drift Handling

Combine scheduled sync with repository restrictions for granular control:

deployment-settings.yml

# Aggressive drift prevention for production
restrictedRepos:
  include:
    - prod-*
  exclude:
    - prod-legacy  # Manual management

.env

# Frequent sync for production repos only
CRON=*/15 * * * *

Temporary Drift Allowance

Temporarily disable drift prevention for maintenance:

deployment-settings.yml

restrictedRepos:
  exclude:
    - admin
    - .github
    - safe-settings
    - api-service  # TODO: Remove after migration (2024-03-20)

Get Started

Deployment

Configuration

Settings Reference

Advanced

Operations

​What is Configuration Drift?

​Drift Prevention Strategies

​Webhook-Based Drift Prevention

​Monitored Events

​Branch Protection Changes

​Repository Settings Changes

​Team Permission Changes

​Ruleset Changes

​Custom Property Changes

​How Webhook Drift Prevention Works

​Configuration for Webhooks

​Webhook Event Examples

​Example 1: Branch Protection Drift

​Example 2: Team Permission Drift

​Scheduled Synchronization

​Why Scheduled Sync?

​Configuring Scheduled Sync

​Cron Syntax

​Common Cron Patterns

​How Scheduled Sync Works

​Scheduled Sync Behavior

​Performance Considerations

​Comparison Logic

​What Gets Compared

​How Comparison Works

​Smart Comparison Features

​Monitoring Drift

​Check Runs

​Check Run Details

​Logging

​Best Practices

​Choose Appropriate Sync Frequency

​Consider API Rate Limits

​Schedule During Off-Peak Hours

​Combine Both Approaches

​Monitor Drift Patterns

​Handle Legitimate Changes

​Troubleshooting

​Drift Not Being Corrected

​Scheduled Sync Not Running

​Too Many API Calls

​Unexpected Drift Corrections

​Advanced Configuration

​Repository-Specific Drift Handling

​Temporary Drift Allowance

​Related Documentation

Build docs developers (and LLMs) love

What is Configuration Drift?

Drift Prevention Strategies

Webhook-Based Drift Prevention

Monitored Events

Branch Protection Changes

Repository Settings Changes

Team Permission Changes

Ruleset Changes

Custom Property Changes

How Webhook Drift Prevention Works

Configuration for Webhooks

Webhook Event Examples

Example 1: Branch Protection Drift

Example 2: Team Permission Drift

Scheduled Synchronization

Why Scheduled Sync?

Configuring Scheduled Sync

Cron Syntax

Common Cron Patterns

How Scheduled Sync Works

Scheduled Sync Behavior

Performance Considerations

Comparison Logic

What Gets Compared

How Comparison Works

Smart Comparison Features

Monitoring Drift

Check Runs

Check Run Details

Logging

Best Practices

Choose Appropriate Sync Frequency

Consider API Rate Limits

Schedule During Off-Peak Hours

Combine Both Approaches

Monitor Drift Patterns

Handle Legitimate Changes

Troubleshooting

Drift Not Being Corrected

Scheduled Sync Not Running

Too Many API Calls

Unexpected Drift Corrections

Advanced Configuration

Repository-Specific Drift Handling

Temporary Drift Allowance

Related Documentation