Production Deployment - Alliance IGAD Innovation Hub

Production deployment requires manual confirmation and affects the live system. Always deploy to testing first and verify functionality.

Prerequisites

Before deploying to production:

Deploy to testing environment and verify all functionality

Run full test suite and ensure all tests pass

Review CloudWatch logs for errors in testing

Obtain approval from stakeholders

Notify users of planned maintenance (if applicable)

Back up critical data if needed

Pre-deployment Checklist

Code Quality

cd igad-app/backend
make all-checks  # Format, lint, type-check, test

cd ../frontend
npm run lint
npm run type-check
npm run test

Testing Environment Validation

# Verify testing deployment is working
curl https://<testing-api>.execute-api.us-east-1.amazonaws.com/prod/api/health

# Check CloudWatch logs for errors
aws logs tail /aws/lambda/igad-backend-testing-ApiFunction \
  --since 1h \
  --profile IBD-DEV

Configuration Review

Verify production configuration in config/production.json:

{
  "environment": "production",
  "lambda": {
    "memorySize": 256,
    "timeout": 30
  },
  "dynamodb": {
    "billingMode": "ON_DEMAND",
    "pointInTimeRecovery": true
  },
  "cognito": {
    "passwordPolicy": {
      "minLength": 12,
      "requireLowercase": true,
      "requireUppercase": true,
      "requireDigits": true,
      "requireSymbols": true
    },
    "mfaConfiguration": "OPTIONAL"
  },
  "s3": {
    "versioning": true,
    "encryption": "S3_MANAGED"
  },
  "cloudwatch": {
    "logRetention": 30
  }
}

Production Deployment Scripts

Full Stack Production Deployment

Script: deploy-fullstack-production.sh Deploys both frontend and backend to production with safety checks.

cd igad-app
./scripts/deploy-fullstack-production.sh

See scripts/deploy-fullstack-production.sh:1

Deployment Process

Step 1: Safety Confirmation

🚀 IGAD Innovation Hub - Fullstack Production Deployment
=======================================================

✅ AWS profile and region validated
   Profile: IBD-DEV
   Region: us-east-1

⚠️  PRODUCTION DEPLOYMENT WARNING ⚠️
This will deploy to PRODUCTION environment.
Ensure all testing has been completed.

Are you sure you want to proceed? (yes/no):

Type yes to continue, or anything else to cancel.

Step 2: Automated Testing

🧪 Running tests...
pytest backend/tests/ -v --tb=short

The deployment will abort if any tests fail. Fix failing tests before retrying.

Step 3: Frontend Build (Production Mode)

🔨 Building frontend for production...
npm --prefix frontend install
npm --prefix frontend run build

Production build settings:

Minification enabled
Source maps generated
Environment variables from .env.production

Step 4: Backend Build

🔨 Building backend...
rm -rf backend/dist
mkdir -p backend/dist
cp -r backend/app backend/dist/
cp backend/requirements.txt backend/dist/
cp backend/bootstrap backend/dist/
cp backend/.env backend/dist/
pip3 install -r backend/requirements.txt -t backend/dist/

Step 5: SAM Deployment

🚀 Deploying to production...
sam build --use-container
sam deploy --config-env production

Uses production configuration from samconfig.toml (if defined) or defaults to:

[production]
[production.deploy]
[production.deploy.parameters]
stack_name = "igad-backend-production"
resolve_s3 = true
confirm_changeset = false
capabilities = "CAPABILITY_IAM"
region = "us-east-1"

Step 6: Resource Discovery

🔍 Finding S3 bucket for production environment...
📤 S3 Bucket: igad-production-websitebucket-xyz789

🔍 Finding CloudFront distribution for S3 bucket...
📤 CloudFront Distribution ID: E0987654321XYZ

Dynamically discovers production resources from CloudFormation stack.

Step 7: Frontend Upload

📤 Uploading frontend to S3...
aws s3 sync frontend/dist/ s3://${BUCKET_NAME} --delete

--delete flag removes old files not in the new build.

Step 8: CloudFront Invalidation

🔄 Invalidating CloudFront cache...
aws cloudfront create-invalidation \
  --distribution-id E0987654321XYZ \
  --paths "/*"

✅ CloudFront invalidation created: I0987654321XYZ

Step 9: Deployment Summary

🎉 Production deployment completed successfully!

✅ Production deployment ready!
📋 Production Resources:
   - Frontend: CloudFront Distribution
   - Backend: Lambda + API Gateway
   - Database: DynamoDB (production table)
   - Auth: Cognito (production pool)

🔍 Post-deployment checklist:
   □ Verify all endpoints respond
   □ Test authentication flow
   □ Check CloudWatch logs
   □ Monitor error rates
   □ Update DNS if using custom domain

Legacy Production Deployment

Script: deploy-production.sh (older version)

./scripts/deploy-production.sh

See scripts/deploy-production.sh:1 Differences:

No automated testing
Backend-only deployment
Manual frontend deployment

Usage:

cd igad-app

# Copy source to dist
cp -r backend/app/* backend/dist/

# Build and deploy
sam build --use-container
sam deploy --config-env production

# Get CloudFront ID from stack outputs
DISTRIBUTION_ID=$(aws cloudformation describe-stacks \
  --stack-name igad-backend-production \
  --query 'Stacks[0].Outputs[?OutputKey==`CloudFrontDistributionId`].OutputValue' \
  --output text)

# Upload frontend
aws s3 sync frontend/dist/ s3://${BUCKET_NAME} --delete

# Invalidate cache
aws cloudfront create-invalidation \
  --distribution-id $DISTRIBUTION_ID \
  --paths "/*"

Post-Deployment Verification

Verify CloudFormation Stack

aws cloudformation describe-stacks \
  --stack-name igad-backend-production \
  --profile IBD-DEV

Check for:

Stack status: CREATE_COMPLETE or UPDATE_COMPLETE
No failed resources
All outputs present

Health Check Endpoints

API Health

curl https://<production-api>.execute-api.us-east-1.amazonaws.com/prod/api/health

Expected response:

{
  "status": "healthy",
  "environment": "production",
  "timestamp": "2026-03-04T12:00:00Z"
}

Vector Storage Health

curl -X POST https://<production-api>.execute-api.us-east-1.amazonaws.com/prod/api/vectors/health \
  -H "Authorization: Bearer <token>"

Expected response:

{
  "status": "operational",
  "bucket": "igad-proposals-vectors-production",
  "indexes": [
    "reference-proposals-index",
    "existing-work-index"
  ]
}

Test Critical User Flows

Authentication

Test user login, password reset, and token refresh

Proposal Creation

Create a test proposal and verify database writes

Document Upload

Upload a document and verify S3 storage

AI Generation

Trigger AI analysis and verify Bedrock integration

Document Download

Download a generated document and verify S3 retrieval

Monitor CloudWatch Logs

Lambda Function Logs

# Main API function
aws logs tail /aws/lambda/igad-backend-production-ApiFunction \
  --follow \
  --profile IBD-DEV

# Worker function
aws logs tail /aws/lambda/igad-backend-production-AnalysisWorkerFunction \
  --follow \
  --profile IBD-DEV

Search for Errors

aws logs filter-log-events \
  --log-group-name /aws/lambda/igad-backend-production-ApiFunction \
  --filter-pattern "ERROR" \
  --start-time $(date -u -d '1 hour ago' +%s)000 \
  --profile IBD-DEV

Check CloudWatch Metrics

# Lambda invocations
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Invocations \
  --dimensions Name=FunctionName,Value=igad-backend-production-ApiFunction \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Sum \
  --profile IBD-DEV

# Lambda errors
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Errors \
  --dimensions Name=FunctionName,Value=igad-backend-production-ApiFunction \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Sum \
  --profile IBD-DEV

Rollback Procedures

CloudFormation Stack Rollback

If deployment fails, CloudFormation automatically rolls back. To manually rollback:

# Get previous stack version
aws cloudformation describe-stacks \
  --stack-name igad-backend-production \
  --profile IBD-DEV \
  --query 'Stacks[0].PreviousStackPolicy'

# Rollback to previous version
aws cloudformation rollback-stack \
  --stack-name igad-backend-production \
  --profile IBD-DEV

Frontend Rollback

S3 versioning is enabled in production. To restore previous frontend:

# List object versions
aws s3api list-object-versions \
  --bucket igad-production-websitebucket-xyz789 \
  --prefix index.html \
  --profile IBD-DEV

# Copy previous version
aws s3api copy-object \
  --bucket igad-production-websitebucket-xyz789 \
  --copy-source igad-production-websitebucket-xyz789/index.html?versionId=<version-id> \
  --key index.html \
  --profile IBD-DEV

# Invalidate CloudFront
aws cloudfront create-invalidation \
  --distribution-id E0987654321XYZ \
  --paths "/index.html" \
  --profile IBD-DEV

Lambda Function Rollback

Revert to a previous Lambda version:

# List function versions
aws lambda list-versions-by-function \
  --function-name igad-backend-production-ApiFunction \
  --profile IBD-DEV

# Update alias to previous version
aws lambda update-alias \
  --function-name igad-backend-production-ApiFunction \
  --name production \
  --function-version <previous-version> \
  --profile IBD-DEV

Production-Specific Configuration

Environment Variables

Production Lambda functions use these environment variables:

Environment:
  Variables:
    PORT: 8080
    AWS_LAMBDA_EXEC_WRAPPER: /opt/bootstrap
    ENVIRONMENT: production
    TABLE_NAME: igad-prod-main-table
    PROPOSALS_BUCKET: !Ref ProposalDocumentsBucket
    COGNITO_CLIENT_ID: <production-client-id>
    COGNITO_USER_POOL_ID: <production-pool-id>
    CORS_ALLOWED_ORIGINS: https://igad-innovation-hub.com,https://www.igad-innovation-hub.com

Cognito Production Settings

Production uses stricter security:

Password Policy: Minimum 12 characters, requires symbols
MFA: Optional (users can enable)
Email Delivery: Cognito default or SES custom sender
Token Validity: Shorter refresh token lifetime

Configure email templates:

python3 backend/scripts/deployment/deploy_production_emails.py

See backend/scripts/deployment/README_EMAIL_DEPLOYMENT.md:1

DynamoDB Production Settings

Billing Mode: On-demand (auto-scaling)
Point-in-Time Recovery: Enabled
Encryption: AWS managed keys
Backup: Daily automated backups (30-day retention)

Monitoring and Alerts

CloudWatch Alarms

Set up alarms for production monitoring:

# Lambda error rate alarm
aws cloudwatch put-metric-alarm \
  --alarm-name igad-production-lambda-errors \
  --alarm-description "Alert on Lambda errors" \
  --metric-name Errors \
  --namespace AWS/Lambda \
  --statistic Sum \
  --period 300 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=FunctionName,Value=igad-backend-production-ApiFunction \
  --profile IBD-DEV

# API Gateway 5xx errors
aws cloudwatch put-metric-alarm \
  --alarm-name igad-production-api-5xx \
  --alarm-description "Alert on API Gateway 5xx errors" \
  --metric-name 5XXError \
  --namespace AWS/ApiGateway \
  --statistic Sum \
  --period 300 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --profile IBD-DEV

Log Insights Queries

Most common errors:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by @message
| sort count desc
| limit 10

Slowest API requests:

fields @timestamp, @duration, @message
| filter @type = "REPORT"
| sort @duration desc
| limit 20

Troubleshooting Production Issues

High Lambda Duration

Symptom: Requests timing out Investigation:

# Check average duration
aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Duration \
  --dimensions Name=FunctionName,Value=igad-backend-production-ApiFunction \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 300 \
  --statistics Average,Maximum \
  --profile IBD-DEV

Solutions:

Increase Lambda memory in template.yaml:28 (memory affects CPU)
Optimize database queries (use indexes)
Enable Lambda provisioned concurrency for warm starts

DynamoDB Throttling

Symptom: ProvisionedThroughputExceededException Solution: DynamoDB is on-demand, but check for hot partitions:

aws dynamodb describe-table \
  --table-name igad-prod-main-table \
  --profile IBD-DEV

Review partition key design if consistent throttling occurs.

Cognito Authentication Failures

Symptom: Users cannot log in Investigation:

# Check Cognito logs (if enabled)
aws cognito-idp admin-list-user-auth-events \
  --user-pool-id <production-pool-id> \
  --username [email protected] \
  --profile IBD-DEV

Common causes:

Incorrect COGNITO_CLIENT_ID or COGNITO_USER_POOL_ID
CORS issues (check allowed origins)
Token expiration (check token validity settings)

Next Steps

Environment Configuration

Configure environment variables and AWS resources

Deployment Overview

Review deployment architecture and AWS services

Get Started

Core Features

Development

Deployment

​Prerequisites

​Pre-deployment Checklist

​Code Quality

​Testing Environment Validation

​Configuration Review

​Production Deployment Scripts

​Full Stack Production Deployment

​Deployment Process

​Step 1: Safety Confirmation

​Step 2: Automated Testing

​Step 3: Frontend Build (Production Mode)

​Step 4: Backend Build

​Step 5: SAM Deployment

​Step 6: Resource Discovery

​Step 7: Frontend Upload

​Step 8: CloudFront Invalidation

​Step 9: Deployment Summary

​Legacy Production Deployment

​Post-Deployment Verification

​Verify CloudFormation Stack

​Health Check Endpoints

​API Health

​Vector Storage Health

​Test Critical User Flows

​Monitor CloudWatch Logs

​Lambda Function Logs

​Search for Errors

​Check CloudWatch Metrics

​Rollback Procedures

​CloudFormation Stack Rollback

​Frontend Rollback

​Lambda Function Rollback

​Production-Specific Configuration

​Environment Variables

​Cognito Production Settings

​DynamoDB Production Settings

​Monitoring and Alerts

​CloudWatch Alarms

​Log Insights Queries

​Troubleshooting Production Issues

​High Lambda Duration

​DynamoDB Throttling

​Cognito Authentication Failures

​Next Steps

Environment Configuration

Deployment Overview

Build docs developers (and LLMs) love

Prerequisites

Pre-deployment Checklist

Code Quality

Testing Environment Validation

Configuration Review

Production Deployment Scripts

Full Stack Production Deployment

Deployment Process

Step 1: Safety Confirmation

Step 2: Automated Testing

Step 3: Frontend Build (Production Mode)

Step 4: Backend Build

Step 5: SAM Deployment

Step 6: Resource Discovery

Step 7: Frontend Upload

Step 8: CloudFront Invalidation

Step 9: Deployment Summary

Legacy Production Deployment

Post-Deployment Verification

Verify CloudFormation Stack

Health Check Endpoints

API Health

Vector Storage Health

Test Critical User Flows

Monitor CloudWatch Logs

Lambda Function Logs

Search for Errors

Check CloudWatch Metrics

Rollback Procedures

CloudFormation Stack Rollback

Frontend Rollback

Lambda Function Rollback

Production-Specific Configuration

Environment Variables

Cognito Production Settings

DynamoDB Production Settings

Monitoring and Alerts

CloudWatch Alarms

Log Insights Queries

Troubleshooting Production Issues

High Lambda Duration

DynamoDB Throttling

Cognito Authentication Failures

Next Steps