Error Handling Strategy
Safe Settings uses different error handling approaches depending on the execution mode:NOP Mode (Dry-Run)
When running in NOP mode (during PR validation), errors are collected and reported without stopping execution:- Continues processing: Other repositories are still validated even if one fails
- Collects all errors: Builds a comprehensive error report
- Creates NopCommand objects: Structured error representation
- Updates PR check: Reports all errors in the check run
lib/settings.js:36-44
Live Mode
When running in live mode (processing actual changes):- Throws exceptions: Failures halt processing for the affected operation
- Creates check runs: Reports success or failure in admin repo
- Logs errors: Detailed error messages for debugging
index.js:35-48
Error Types
Configuration Errors
What causes them:- Invalid YAML syntax in settings files
- Missing required fields in configuration
- Incorrect data types (e.g., string instead of number)
- Deployment config file not found
- Caught during config loading
- Logged with full error details
- Reported in check run as validation failure
- NopCommand created with ERROR type
lib/settings.js:554-593
Validation Errors
What causes them:- Custom validation rules fail (
configvalidators) - Override validation rules fail (
overridevalidators) - Settings violate organizational policies
- Validation functions return
false - Error message from validator is thrown
- Processing stops for that configuration
- Detailed error appears in check run
lib/settings.js:476-493
API Errors
What causes them:- GitHub API rate limiting
- Insufficient permissions
- Network connectivity issues
- Invalid API requests
- Resources not found (404 errors)
- Probot automatically retries rate-limited requests
- Errors are logged with full details
- 404 errors often indicate missing resources (expected in some cases)
- Check runs report API failures
lib/settings.js:555-592
Repository Access Errors
What causes them:- Repository doesn’t exist
- App doesn’t have access to repository
- Admin repository not found
- Insufficient permissions
- 404 errors are caught and logged
- Graceful degradation (skips inaccessible repos)
- Reports error in check run
lib/settings.js:166-173
Error Reporting Mechanisms
Check Runs in Admin Repository
After every sync operation (whether triggered by webhook or cron), Safe Settings creates a check run in the admin repository.
Source:
lib/settings.js:129-174
Check Run Output Structure
Check runs include:- Title: “Safe-Settings” or “Safe-Settings Dry-Run Finished with Error/success”
- Summary: High-level status message
- Details: Rendered from error template with:
- Repository name
- Plugin that failed
- Error message
- Additions/modifications/deletions attempted
lib/settings.js:129-174
NopCommand Structure
Errors in NOP mode are represented asNopCommand objects:
Source: lib/nopcommand.js:1-21
Logging
All errors are logged using the Probot logger: Source:lib/settings.js:176-184
PR Validation Error Handling
When validating configuration changes in pull requests, Safe Settings provides detailed feedback.Validation Process
index.js:541-613
PR Check Output
The check run includes:lib/settings.js:262-294
Validation Error Example
README.md:246-263
Error Recovery
Automatic Recovery
- Scheduled sync: Automatically retries failed operations on next sync
- Webhook retry: GitHub automatically retries failed webhook deliveries
- Rate limit handling: Probot automatically waits and retries
Manual Recovery
Fix configuration errors
Fix configuration errors
- Review error message in check run
- Correct the YAML syntax or configuration
- Commit the fix to the admin repository
- Safe Settings automatically processes the corrected config
Address validation failures
Address validation failures
- Review the validation error message
- Either:
- Adjust the configuration to meet validation rules
- Update the validation rule if requirements changed
- Commit and push the changes
Handle API permission errors
Handle API permission errors
- Verify GitHub App has required permissions
- Check that repositories are accessible
- Ensure admin repository exists and is accessible
- Reinstall GitHub App if necessary
Resolve rate limiting
Resolve rate limiting
- Reduce sync frequency (adjust CRON schedule)
- Optimize configuration to minimize API calls
- Consider GitHub Enterprise with higher rate limits
- Wait for rate limit to reset (typically 1 hour)
Debugging Errors
Enable Debug Logging
.env
README.md:524-527
Check Run Details
Click on failed check runs in the admin repository to view:- Full error messages
- Stack traces (in debug mode)
- API endpoints called
- Request/response bodies
Review Logs
Check your deployment logs for:- Error stack traces
- API call details
- Configuration loading issues
- Validation failures
Common Issues
Error: Admin Repo Not found
Error: Admin Repo Not found
Cause: The admin repository doesn’t exist or the app doesn’t have access.Solution:
- Verify admin repository exists
- Check
ADMIN_REPOenvironment variable - Ensure GitHub App is installed on the organization
- Verify repository permissions
Error: YAML parsing failed
Error: YAML parsing failed
Cause: Invalid YAML syntax in configuration file.Solution:
- Use a YAML validator to check syntax
- Check for correct indentation (spaces, not tabs)
- Ensure proper quoting of special characters
- Validate with online YAML linters
Error: Multiple suborg configs for repository
Error: Multiple suborg configs for repository
Cause: Repository matches multiple suborg configuration patterns.Solution:
- Review suborg configurations for overlaps
- Make patterns more specific
- Use
excludeto prevent conflicts - Ensure repository only belongs to one suborg
lib/settings.js:824-829Error: Validation failed
Error: Validation failed
Cause: Custom validation rule rejected the configuration.Solution:
- Read the validation error message carefully
- Review the validation script in
deployment-settings.yml - Adjust configuration to meet validation requirements
- Update validation rules if they’re too restrictive
Best Practices
- Use PR validation: Always create PRs for config changes to catch errors before they’re applied
- Test with small changes: Make incremental changes rather than large batch updates
- Monitor check runs: Regularly review check run results in the admin repository
- Enable PR comments: Set
ENABLE_PR_COMMENT=truefor detailed validation feedback - Set appropriate log levels: Use
debugduring setup,infoin production - Document custom validators: Clearly document validation rules and their purpose
- Handle 404 gracefully: Some 404 errors are expected (e.g., missing repo configs)
- Review logs regularly: Check logs for patterns of errors or warnings