Overview
Authentication errors can prevent users from logging in or registering, causing critical service disruptions. The most common issues include:- Type mismatches between bcrypt library versions
- Missing authentication middleware
- Password hash encoding/decoding errors
- JWT token generation failures
Incidents
INC-001: Login endpoint returns 500 after bcrypt upgrade
INC-001: Login endpoint returns 500 after bcrypt upgrade
Incident Details
- Severity: P1 - Critical
- Service: python-service
- Environment: Staging
- Reported: 2026-02-28T14:23:00Z
- Status: Resolved
Problem
After upgrading bcrypt from version 3.2.0 to 4.1.2, the/api/auth/login endpoint started returning 500 Internal Server Error for all login attempts, even with valid credentials. The endpoint was working correctly before the library upgrade.Error Message:Root Cause
The bcrypt library changed its API in version 4.0.0. Thecheckpw() function now requires the stored password hash to be in bytes format, but the application was storing password hashes as strings in the database and passing strings to bcrypt.checkpw().Location: app/routes/auth.py:53-56Problematic Code
auth.py
Resolution
Encode the stored password hash to bytes before passing it tobcrypt.checkpw():auth.py
Prevention
- Test library upgrades: Always test dependency upgrades in a staging environment before production deployment
- Review changelogs: Check breaking changes in major version updates (3.x → 4.x)
- Type consistency: Store password hashes as bytes in the database to match bcrypt’s expected input type
- Add integration tests: Create tests that verify authentication flows work end-to-end
Related Issues
- Bcrypt 4.0.0 breaking changes: https://github.com/pyca/bcrypt/blob/main/CHANGELOG.rst
- Similar issue in registration endpoint (also requires
.encode()for hash storage)
INC-002: Database connection failure in staging/production
INC-002: Database connection failure in staging/production
Incident Details
- Severity: P1 - Critical
- Service: python-service
- Environment: Staging
- Reported: 2026-02-28T09:15:00Z
- Status: Resolved
Problem
The Python service failed to connect to PostgreSQL in staging and production environments. The application started successfully but crashed immediately when any database-backed endpoint was accessed. Local development worked fine, but deployed environments couldn’t connect to the database.Error Message:Root Cause
The development configuration used different environment variable names (DATABASE_USER, DATABASE_PASSWORD, DATABASE_HOST) than the staging/production configurations (DB_USER, DB_PASS, DB_HOST). When the infrastructure team standardized environment variables to use the DB_* prefix, the development config still had hardcoded fallbacks to localhost, causing staging/production to fall back to incorrect defaults.Location: app/config.py:11-32Problematic Code
config.py
Resolution
Standardize environment variable names across all configurations and remove unsafe fallback defaults:config.py
Prevention
- Consistent naming: Use the same environment variable names across all environments
- Fail fast: Don’t use fallback defaults for critical configuration in production environments
- Environment validation: Add startup checks to validate required environment variables are set
- Documentation: Document all required environment variables in README or deployment docs
- Configuration testing: Test configuration loading in CI for each environment
INC-003: ImportError after Flask upgrade to 2.3.x
INC-003: ImportError after Flask upgrade to 2.3.x
Incident Details
- Severity: P2 - High
- Service: python-service
- Environment: All
- Reported: 2026-02-27T16:45:00Z
- Status: Resolved
Problem
After upgrading Flask from 2.2.x to 2.3.x for security patches, the application failed to start with an ImportError. The custom JSON encoder that handles datetime and Decimal serialization broke because Flask removed theflask.json.JSONEncoder class in version 2.3.Error Message:Root Cause
Flask 2.3.0 removed theflask.json.JSONEncoder class as part of a refactoring to use Python’s built-in json module more directly. The application’s custom JSON encoder inherited from the removed class.Location: app/__init__.py:6,15-23,34Problematic Code
__init__.py
Resolution
Use Flask’s new JSON provider interface introduced in Flask 2.2:__init__.py
Prevention
- Read upgrade guides: Review Flask’s migration guide when upgrading major/minor versions
- Check deprecation warnings: Run application with deprecation warnings enabled during testing
- Pin major versions: Use version constraints like
Flask>=2.3,<3.0to avoid surprise breaking changes - Test upgrades: Run full test suite after any dependency upgrade
- Gradual adoption: Upgrade dependencies one at a time to isolate issues
Related Documentation
- Flask 2.3 changelog: https://flask.palletsprojects.com/en/latest/changes/#version-2-3-0
- Custom JSON provider guide: https://flask.palletsprojects.com/en/latest/api/#flask.json.provider.DefaultJSONProvider
Common Patterns
Password Hashing Best Practices
Environment Variable Best Practices
Quick Reference
| Issue Type | Common Cause | Quick Fix |
|---|---|---|
| bcrypt TypeError | Library version mismatch | Add .encode('utf-8') to hash parameter |
| Database connection fails | Wrong environment variables | Standardize variable names, validate on startup |
| ImportError on Flask upgrade | Deprecated/removed API | Use new DefaultJSONProvider interface |
| JWT token invalid | Wrong secret key | Verify JWT_SECRET_KEY is set correctly |