Overview
The GovTech platform uses two deployment workflows:- Deploy to Dev - Automatic deployment to development environment
- Deploy to Production - Manual blue-green deployment with approval gates
Deployment Strategy
Deploy to Dev Workflow
File:.github/workflows/deploy-dev.yml
Purpose
Automatically deploy to development environment when code is pushed tostaging branch.
Trigger
Environment
dev (can configure secrets and protection rules)
Job: Deploy
Full workflow: deploy-dev.yml:10-91Step 1: AWS Authentication
Step 2: Terraform Infrastructure Provisioning
Setup Terraform:-
Initialize (deploy-dev.yml:34-36):
Downloads providers and modules.
-
Plan (deploy-dev.yml:38-43):
Generates execution plan with changes. Environment Variables:
-
Apply (deploy-dev.yml:45-47):
Creates/updates infrastructure automatically.
./terraform/environments/dev
What Terraform Provisions:
- EKS cluster (
govtech-dev) - RDS database
- VPC and networking
- IAM roles and policies
- ECR repositories
Step 3: Kubernetes Deployment
Setup kubectl:- Namespace:
govtech - ConfigMap: Application configuration
- Backend Deployment and Service
- Frontend Deployment and Service
Step 4: Wait for Rollout
- Waits up to 5 minutes for pods to be ready
- Fails deployment if rollout doesn’t complete
- Ensures zero-downtime deployment
Step 5: Smoke Tests
Dev Deployment Timeline
Total Time: ~3-4 minutesDeploy to Production Workflow
File:.github/workflows/deploy-prod.yml
Purpose
Manual blue-green deployment to production environment with validation and rollback capability.Trigger
version: Docker image tag (SHA orlatest)confirm: Must typeDEPLOYto proceed
Jobs
Job 1: Validate
Purpose: Prevent accidental deployments.- Fails immediately if user doesn’t type “DEPLOY”
- Prevents typos or accidental clicks
Job 2: Deploy
Dependencies:environment: production triggers GitHub Environment protection rules:
- Required reviewers (manual approval)
- Deployment branch restrictions
- Environment secrets
Blue-Green Deployment Strategy
Concept: Steps:- Deploy GREEN (new version) alongside existing BLUE
- Wait for health checks to pass on GREEN
- Switch traffic from BLUE to GREEN
- Keep BLUE running for instant rollback if needed
Deploy GREEN Version
$IMAGE_TAG: From workflow input (e.g.,abc123456)--record: Save command in revision history for rollback-n govtech: Namespace
- Creates new ReplicaSet with new image
- Starts new pods (GREEN)
- Old pods (BLUE) remain running
- Traffic stays on BLUE until new pods are ready
Wait for GREEN Readiness
- Pods pass readiness probes
- All replicas are ready
- No CrashLoopBackOff errors
Production Health Checks
- Backend must have at least 2 ready replicas
- Frontend must be running
- Fails deployment if health checks don’t pass
Traffic Switch
Kubernetes automatically switches traffic when:- New pods (GREEN) pass readiness probes
kubectl rollout statuscompletes successfully- Service selector matches new pods
Deployment Notification
Job 3: Rollback
Purpose: Automatic rollback if deployment fails.deploy job fails.
Rollback Process:
- Reverts to previous deployment revision
- Uses Kubernetes rollout history (
--recordfrom deploy step) - Waits for rollback to complete
- Restores BLUE version automatically
Deployment Comparison
| Feature | Deploy Dev | Deploy Prod |
|---|---|---|
| Trigger | Automatic (push to staging) | Manual (workflow_dispatch) |
| Approval | None | GitHub Environment reviewers |
| Terraform | Yes (auto-approve) | No (infrastructure pre-provisioned) |
| Strategy | Rolling update | Blue-green |
| Rollback | Manual | Automatic on failure |
| Timeout | 5 minutes | 10 minutes |
| Health Checks | Basic smoke tests | Replica count validation |
| Environment | govtech-dev EKS | govtech-prod EKS |
Environment Configuration
GitHub Environments
Configure in: Repository Settings > EnvironmentsDev Environment
Production Environment
Required Secrets
| Secret | Environment | Purpose |
|---|---|---|
AWS_ACCESS_KEY_ID | dev, production | AWS authentication |
AWS_SECRET_ACCESS_KEY | dev, production | AWS authentication |
DB_PASSWORD | dev | RDS database password |
DB_PASSWORD_PROD | production | Production database password |
AWS_DEPLOY_ROLE_ARN | CI workflows | OIDC role ARN |
Kubernetes Resources Deployed
Namespace
File:kubernetes/namespace.yaml
ConfigMap
File:kubernetes/configmap.yaml
Contains:
- API endpoints
- Feature flags
- Non-sensitive configuration
Backend Deployment
File:kubernetes/backend/deployment.yaml
Key settings:
Frontend Deployment
File:kubernetes/frontend/deployment.yaml
Serves static assets via Nginx.
Services
Expose deployments:- Backend:
ClusterIPservice on port 8080 - Frontend:
LoadBalancerservice on port 80
Deployment Timeline
Development Deployment
Production Deployment
Best Practices
Pre-deployment Checklist
- All CI checks passed
- Security scans passed
- Code reviewed and approved
- Database migrations tested
- Rollback plan documented
During Deployment
- Monitor pod status:
kubectl get pods -n govtech -w - Check logs:
kubectl logs -f deployment/backend -n govtech - Verify metrics in monitoring dashboards
- Test critical user flows
Post-deployment
- Verify application functionality
- Check error rates in APM
- Review deployment logs
- Update deployment documentation
- Notify stakeholders
Rollback Procedures
Automatic Rollback (if deployment fails):Monitoring Deployments
GitHub Actions UI
View Workflow:- Go to Actions tab
- Select workflow run
- View job logs and status
- Navigate to pending deployment
- Click Review deployments
- Select production environment
- Click Approve and deploy
kubectl Commands
AWS EKS Console
View Cluster:- AWS Console > EKS
- Select
govtech-devorgovtech-prod - View Workloads tab
- Monitor pod health
Troubleshooting
Deployment Stuck on Rollout
Symptom:kubectl rollout status times out.
Diagnosis:
- Image pull errors (wrong tag or ECR permissions)
- Application crashes (check logs)
- Failed readiness probes (check probe configuration)
- Resource limits (insufficient CPU/memory)
Terraform Apply Fails
Symptom: Terraform apply exits with error. Diagnosis:- Conflicting resource changes
- Missing AWS permissions
- State file lock
Rollback Fails
Symptom:kubectl rollout undo doesn’t restore service.
Diagnosis:
Security Considerations
Deployment Approvals
Production deployments require:- Manual confirmation (type “DEPLOY”)
- GitHub Environment reviewer approval
- Minimum 2 approvers recommended
Secrets Management
Current: GitHub Secrets Recommended: Migrate to:- AWS Secrets Manager for database passwords
- AWS Systems Manager Parameter Store for configuration
- OIDC for AWS authentication (remove access keys)
Audit Trail
Logged Information:- Who triggered deployment (
${{ github.actor }}) - What version was deployed (
${{ github.event.inputs.version }}) - When deployment occurred (
date -u) - Deployment outcome (success/failure)
- GitHub Actions run history (indefinite)
- Kubernetes audit logs (AWS CloudWatch)
- AWS CloudTrail (API calls)