Performance Optimization

Safe Settings is designed to manage organizations with thousands of repositories efficiently. This page explains the performance optimizations that allow Safe Settings to scale.

Performance Challenges

When managing large organizations, Safe Settings faces several constraints:

GitHub App Token Lifetime

GitHub App tokens expire after 1 hour
All synchronization work must complete within this window
Token renewal isn’t always possible during long-running operations

API Rate Limits

GitHub enforces rate limits on API calls
Standard rate limit: 5,000 requests per hour per installation
Making unnecessary API calls consumes quota quickly

Configuration Loading

Loading thousands of repo configuration files is expensive
Parsing and merging configurations takes time
Network latency adds up across many files

Change Detection

Comparing configurations with actual GitHub state for every repo
Deep object comparison is computationally expensive
Generating detailed diff reports requires processing

Optimization Strategies

Safe Settings uses multiple strategies to address these challenges:

1. Selective Configuration Loading

Instead of loading all configuration files for every event, Safe Settings intelligently determines which files are needed.

How It Works

# Change in .github/settings.yml
# (affects all repositories)

Action: Load ALL configuration files
  - .github/settings.yml
  - .github/suborgs/*.yml (all suborg files)
  - .github/repos/*.yml (all repo files)

Reason: Global change affects every repository

Implementation

The loadConfigs and getRepoConfigs functions implement selective loading:

// From lib/settings.js

async loadConfigs(repo) {
  this.subOrgConfigs = await this.getSubOrgConfigs()
  this.repoConfigs = await this.getRepoConfigs(repo)  // repo parameter controls scope
}

async getRepoConfigs(repo) {
  const overridePaths = await this.getRepoConfigMap()
  const repoConfigs = {}
  
  for (const override of overridePaths) {
    // Skip if already loaded
    if (repoConfigs[override.name]) continue
    
    // If repo is specified, only load that repo's config
    if (repo) {
      if (override.name === `${repo.repo}.yml` || override.name === `${repo.repo}.yaml`) {
        const data = await this.loadYaml(override.path)
        repoConfigs[override.name] = data
      }
    } else {
      // Load all repo configs for global sync
      const data = await this.loadYaml(override.path)
      repoConfigs[override.name] = data
    }
  }
  
  return repoConfigs
}

Performance Impact

Global change:
  - Load 1 org file
  - Load 10 suborg files
  - Load 1000 repo files
  - Total: 1011 files

Repo change:
  - Load 1 org file
  - Load 10 suborg files
  - Load 1000 repo files
  - Total: 1011 files (same!)

Result: Every change loads all files

2. Intelligent Change Detection

Safe Settings only makes API calls when there are actual differences between configuration and GitHub state.

The compareDeep Function

The compareDeep function generates detailed differences between expected and actual configurations:

// From lib/mergeDeep.js

compareDeep(target, source, additions, modifications, deletions) {
  // Returns:
  // {
  //   additions: {},      // New settings to add
  //   modifications: {},  // Settings to update
  //   deletions: {},      // Settings to remove
  //   hasChanges: true    // Boolean flag
  // }
}

hasChanges Flag

The critical optimization is the hasChanges boolean:

return ({
  additions,
  modifications,
  deletions,
  hasChanges: !this.isEmpty(additions) || 
              !this.isEmpty(modifications) || 
              !this.isEmpty(deletions)
})

Conditional API Calls

Plugins only make API calls when changes are detected:

// From lib/plugins/branches.js

const changes = this.comparator.compareDeep(
  existingBranchProtection,
  desiredBranchProtection
)

if (!changes.hasChanges) {
  // No API call needed - skip
  return
}

// Only reaches here if there are real changes
await this.github.repos.updateBranchProtection({
  owner: this.repo.owner,
  repo: this.repo.repo,
  branch: branch.name,
  ...desiredBranchProtection
})

What Gets Compared

The comparison logic handles complex nested structures:

Expected: { "name": "main" }
Actual:   { "name": "main" }
Result:   hasChanges = false (identical)

Ignored Fields

The comparison intelligently ignores irrelevant fields:

// From lib/mergeDeep.js

for (const key in source) {
  // Skip URLs and other metadata
  if (key.indexOf('url') >= 0 || this.ignorableFields.indexOf(key) >= 0) {
    continue
  }
  // ... comparison logic
}

Ignored fields include:

URL fields (url, users_url, teams_url, etc.)
GitHub-generated IDs
Timestamps
Metadata fields

This prevents false positives that would trigger unnecessary API calls.

3. Automatic Rate Limit Handling

Probot (the framework Safe Settings is built on) automatically handles GitHub’s rate limits.

How Probot Handles Rate Limits

// Automatic handling by Probot:
// 1. Monitors rate limit headers
// 2. Pauses requests when limit approached
// 3. Resumes when rate limit resets
// 4. Queues requests during pause

Abuse Detection

Probot also handles abuse detection:

// When GitHub returns 403 with abuse detection:
// 1. Probot detects abuse flag
// 2. Backs off exponentially
// 3. Retries after delay
// 4. Continues processing

No manual rate limit handling required in Safe Settings code.

Monitoring Rate Limits

# Check current rate limit status
curl -H "Authorization: Bearer $TOKEN" \
  https://api.github.com/rate_limit

# Response:
{
  "resources": {
    "core": {
      "limit": 5000,
      "remaining": 4850,
      "reset": 1710532800
    }
  }
}

4. Optimized Pull Request Checks

Pull request validation runs in dry-run mode with summarized output.

Summary Instead of Details

For large changes affecting many repositories:

## Changes for 1000 repositories

### Repository: repo-1
```json
{
  "branches": [...],
  "teams": [...],
  "collaborators": [...]
}

Disable PR Comments

For very large organizations, disable PR comments entirely:

.env

ENABLE_PR_COMMENT=false

Details are still available in check runs.

Performance Metrics

Time to Process 1000 Repositories

Without optimizations:
  - Load configs: 120 seconds
  - Compare all repos: 300 seconds
  - Apply changes (no hasChanges check): 600 seconds
  - Total: ~17 minutes

With optimizations:
  - Load configs: 120 seconds
  - Compare all repos: 300 seconds
  - Apply only real changes: 60 seconds
  - Total: ~8 minutes

Improvement: 53% faster

API Calls Saved

Scheduled sync (1000 repos):
  - Branch protection: 1000 API calls
  - Team permissions: 1000 API calls
  - Repository settings: 1000 API calls
  - Labels: 1000 API calls
  - Total: 4000 API calls

Even when nothing changed!

Best Practices for Performance

1. Use Appropriate Sync Frequency

Balance drift prevention with performance:

# Every 15 minutes
CRON=*/15 * * * *

# API calls: ~4000/hour (for 1000 repos)
# Token usage: Multiple tokens per hour

2. Leverage Webhook Events

Webhooks are more efficient than scheduled sync:

Webhook event (branch protection changed):
  - Triggered: Immediately
  - Repos affected: 1
  - Config files loaded: 1-3
  - API calls: 1-2
  - Duration: <5 seconds

Scheduled sync:
  - Triggered: Every N hours
  - Repos affected: All (1000s)
  - Config files loaded: Thousands
  - API calls: Dozens to hundreds
  - Duration: Minutes

Recommendation: Use webhooks as primary, scheduled sync as backup

3. Organize Configurations Efficiently

Use suborgs to group related repositories:

.github/
  settings.yml
  repos/
    repo-1.yml
    repo-2.yml
    repo-3.yml
    ... (1000 files)

Problem: Every global change requires loading 1000 files

4. Minimize Repo-Level Overrides

Fewer repo-level files = faster loading:

Anti-pattern:
  - Every repo has a repos/*.yml file
  - Files mostly duplicate suborg settings
  - Result: 1000 files to load and parse

Better:
  - Most repos inherit from suborg
  - Only special cases have repos/*.yml
  - Result: 50 files to load and parse

5. Use Restricted Repos for Phased Rollout

Limit scope during initial deployment:

restrictedRepos:
  include:
    - pilot-*  # 10 repos

Processing time: Seconds
API calls: Minimal

6. Monitor Performance

Regularly review check run durations:

# Check recent sync durations
gh api repos/:owner/admin/commits/:sha/check-runs \
  --jq '.check_runs[] | select(.name=="safe-settings") | {completed_at, started_at}'

# Calculate duration
started=$(date -d "2024-03-15T10:00:00Z" +%s)
completed=$(date -d "2024-03-15T10:08:45Z" +%s)
echo "Duration: $((completed - started)) seconds"

Troubleshooting Performance Issues

Sync Taking Too Long

Problem: Sync doesn’t complete within 1 hour. Diagnosis:

# Check number of repos
gh repo list your-org --limit 10000 | wc -l

# Check config file count
find .github/repos -name '*.yml' | wc -l

Solutions:

Reduce sync frequency

# From hourly to every 6 hours
CRON=0 */6 * * *

Use restricted repos

restrictedRepos:
  include:
    - critical-*  # Manage fewer repos

Consolidate configs

# Move common settings to suborgs
# Reduce number of repo-level files

High API Usage

Problem: Approaching rate limits frequently. Diagnosis:

# Monitor rate limit
watch -n 60 'gh api rate_limit --jq .resources.core'

Solutions:

Verify hasChanges is working

// Add logging
console.log(`hasChanges: ${changes.hasChanges}`)

Reduce sync frequency
Use webhooks instead of scheduled sync
Check for comparison bugs (false positives)

Slow Configuration Loading

Problem: Loading configs takes too long. Diagnosis:

# Count config files
find .github -name '*.yml' -o -name '*.yaml' | wc -l

# Check file sizes
find .github -name '*.yml' -exec wc -l {} + | sort -n

Solutions:

Reduce number of repo configs
Optimize file sizes (remove comments, minimize duplication)
Use suborg configs instead of individual repo configs

Scaling Considerations

Small Organizations (1-100 repos)

Configuration:
  - Mostly global settings
  - Few repo overrides
  - Aggressive sync schedule

Performance:
  - Sync time: Seconds
  - API calls: Minimal
  - No optimization needed

Medium Organizations (100-1000 repos)

Configuration:
  - Suborg organization
  - Selective repo overrides
  - Moderate sync schedule

Performance:
  - Sync time: 1-5 minutes
  - API calls: Moderate
  - Standard optimizations sufficient

Large Organizations (1000+ repos)

Configuration:
  - Heavy suborg usage
  - Minimal repo overrides
  - Conservative sync schedule
  - Restricted repos for phasing

Performance:
  - Sync time: 5-15 minutes
  - API calls: High but optimized
  - All optimizations critical

Future Optimizations

Potential improvements being considered:

Parallel processing: Process multiple repos concurrently
Incremental sync: Track last sync time, only check changed repos
Caching: Cache GitHub state to reduce API calls
Batch API calls: Use GraphQL for bulk operations
Config preprocessing: Pre-compile configuration hierarchies

Get Started

Deployment

Configuration

Settings Reference

Advanced

Operations

​Performance Challenges

​GitHub App Token Lifetime

​API Rate Limits

​Configuration Loading

​Change Detection

​Optimization Strategies

​1. Selective Configuration Loading

​How It Works

​Implementation

​Performance Impact

​2. Intelligent Change Detection

​The compareDeep Function

​hasChanges Flag

​Conditional API Calls

​What Gets Compared

​Ignored Fields

​3. Automatic Rate Limit Handling

​How Probot Handles Rate Limits

​Abuse Detection

​Monitoring Rate Limits

​4. Optimized Pull Request Checks

​Summary Instead of Details

​Disable PR Comments

​Performance Metrics

​Time to Process 1000 Repositories

​API Calls Saved

​Best Practices for Performance

​1. Use Appropriate Sync Frequency

​2. Leverage Webhook Events

​3. Organize Configurations Efficiently

​4. Minimize Repo-Level Overrides

​5. Use Restricted Repos for Phased Rollout

​6. Monitor Performance

​Troubleshooting Performance Issues

​Sync Taking Too Long

​High API Usage

​Slow Configuration Loading

​Scaling Considerations

​Small Organizations (1-100 repos)

​Medium Organizations (100-1000 repos)

​Large Organizations (1000+ repos)

​Future Optimizations

​Related Documentation

Build docs developers (and LLMs) love

Performance Challenges

GitHub App Token Lifetime

API Rate Limits

Configuration Loading

Change Detection

Optimization Strategies

1. Selective Configuration Loading

How It Works

Implementation

Performance Impact

2. Intelligent Change Detection

The compareDeep Function

hasChanges Flag

Conditional API Calls

What Gets Compared

Ignored Fields

3. Automatic Rate Limit Handling

How Probot Handles Rate Limits

Abuse Detection

Monitoring Rate Limits

4. Optimized Pull Request Checks

Summary Instead of Details

Disable PR Comments

Performance Metrics

Time to Process 1000 Repositories

API Calls Saved

Best Practices for Performance

1. Use Appropriate Sync Frequency

2. Leverage Webhook Events

3. Organize Configurations Efficiently

4. Minimize Repo-Level Overrides

5. Use Restricted Repos for Phased Rollout

6. Monitor Performance

Troubleshooting Performance Issues

Sync Taking Too Long

High API Usage

Slow Configuration Loading

Scaling Considerations

Small Organizations (1-100 repos)

Medium Organizations (100-1000 repos)

Large Organizations (1000+ repos)

Future Optimizations

Related Documentation