Deployment Strategies - System Design 101

Why Deployment Strategies Matter

Deploying or upgrading services is risky. The right deployment strategy can minimize downtime, reduce risk, and enable faster recovery from issues. This guide explores various risk mitigation strategies for production deployments. Deployment Strategies

Common Deployment Strategies

Here are the most-used deployment strategies:

Big Bang Deployment (Multi-Service)
Rolling Deployment
Blue-Green Deployment
Canary Deployment
A/B Testing
Feature Toggle

Let’s explore each strategy in detail.

Multi-Service (Big Bang) Deployment

Overview

In this model, we deploy new changes to multiple services simultaneously. All services are upgraded at the same time in one big release.

Characteristics

✓ Simple to implement
✓ Everything updates together
✓ No version compatibility issues
✓ Coordinated release

When to Use

Small applications with few services
Non-critical systems
Development/testing environments
Scheduled maintenance windows

Risk Level: High - All services change at once, making it difficult to isolate issues.

Rolling Deployment

Overview

Application instances are updated one by one or in small batches, ensuring high availability during the process.

Start with Current Version

All instances running version 1.0

Update First Batch

Stop 1-2 instances, update to version 2.0, restart

Verify Health

Check that updated instances are healthy

Continue Rolling

Repeat for remaining instances

Complete Rollout

All instances now running version 2.0

Characteristics

✓ No downtime
✓ Gradual rollout
✓ Resource efficient
✓ Built into Kubernetes

Configuration Example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1      # Max pods unavailable during update
      maxSurge: 2            # Max extra pods during update
  template:
    spec:
      containers:
      - name: app
        image: my-app:2.0

When to Use

Downtime: No
Use case: Periodic releases, regular updates
Best for applications that can handle multiple versions running simultaneously

Blue-Green Deployment

Overview

Two identical environments are maintained:

Blue: Current production version
Green: New version prepared for release

Once testing is done in the green environment, traffic switches from blue to green.

Blue is Production

All traffic goes to the blue environment (v1.0)

Deploy to Green

Deploy new version (v2.0) to green environment

Test Green

Thoroughly test green environment

Switch Traffic

Route all traffic from blue to green

Monitor

Keep blue environment running for quick rollback

Cleanup (Optional)

Once stable, blue becomes the next green

Characteristics

✓ Zero downtime
✓ Instant rollback
✓ Full testing before switch
✓ Clean separation

Implementation Example

# Initially: Blue target group receives 100% traffic

# Step 1: Deploy to Green
apiVersion: v1
kind: Service
metadata:
  name: my-app-green
spec:
  selector:
    app: my-app
    version: v2
  ports:
  - port: 80

# Step 2: Switch traffic via load balancer
# Update ALB listener rule to point to green target group

# Step 3: Blue becomes standby for rollback

When to Use

Downtime: No
Use case: High-stake updates, critical applications
Best when instant rollback is critical
Suitable for applications with sufficient budget for dual infrastructure

Canary Deployment

Overview

The new version is released to a small subset of users or servers for testing before broader deployment. Named after “canary in a coal mine” - an early warning system.

Deploy Canary

Release new version to 5-10% of servers/users

Monitor Metrics

Watch error rates, performance, user feedback

Gradual Increase

If stable, increase to 25%, then 50%, then 75%

Full Rollout

Eventually reach 100% of traffic

Rollback if Needed

Quickly route traffic away from canary if issues detected

Characteristics

✓ Reduced risk
✓ Real production testing
✓ Gradual rollout
✓ Cost effective

Implementation Example

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-app
spec:
  hosts:
  - my-app
  http:
  - match:
    - headers:
        user-type:
          exact: beta
    route:
    - destination:
        host: my-app
        subset: v2
  - route:
    - destination:
        host: my-app
        subset: v1
      weight: 90
    - destination:
        host: my-app
        subset: v2
      weight: 10

Canary Metrics to Monitor

Error rate increase
Response time degradation
CPU/Memory usage
Business metrics (conversion rate, etc.)
User complaints/feedback

When to Use

Downtime: No
Use case: Impact validation on subset of users
Best for risk-averse organizations
Ideal when you have good monitoring and can test in production

Canary deployments are used by Netflix, Facebook, and Google to safely roll out changes to millions of users.

A/B Testing

Overview

Different versions of services run in production simultaneously. Each version runs an “experiment” for a subset of users to compare performance or user experience.

A/B Testing vs Canary

- Focus: Risk mitigation
- Goal: Safe rollout
- Duration: Temporary (hours/days)
- Metrics: Technical (errors, latency)
- Outcome: Replace old version

Implementation Example

apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-flags
data:
  features.json: |
    {
      "new-checkout": {
        "enabled": true,
        "percentage": 50,
        "targeting": {
          "userSegments": ["premium"]
        }
      }
    }

Characteristics

✓ Data-driven decisions
✓ Measure business impact
✓ Multiple variants possible
✓ Cost effective testing

When to Use

Downtime: Not directly applicable
Use case: Optimizing user experience, feature validation
Testing new features or UI changes
Measuring business impact before full rollout

Feature Toggle (Feature Flags)

Overview

Features are deployed to production but hidden behind toggles. They can be enabled/disabled without redeployment.

Deploy with Feature Disabled

Code is deployed but feature is off by default

Enable for Testing

Turn on feature for internal users or test accounts

Gradual Rollout

Enable for increasing percentages of users

Full Release

Enable for all users

Cleanup

Remove flag after stable (technical debt management)

Types of Feature Flags

- Toggle new features on/off
- Short-lived (remove after release)
- Used for gradual rollouts

Implementation Example

const LD = require('launchdarkly-node-server-sdk');

const client = LD.init(process.env.LAUNCHDARKLY_SDK_KEY);

await client.waitForInitialization();

const showNewFeature = await client.variation(
  'new-feature',
  { key: user.id, email: user.email },
  false // default value
);

if (showNewFeature) {
  // Show new feature
} else {
  // Show old feature
}

Best Practices

Name flags clearly: Use descriptive names like enable-new-checkout
Set expiration: Plan to remove flags after rollout
Monitor flag usage: Track which flags are active
Limit flag count: Too many flags create complexity
Document flags: Explain purpose and expected lifecycle

Kubernetes Deployment Strategies

Kubernetes supports several deployment strategies natively:

Recreate Strategy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  strategy:
    type: Recreate
  template:
    spec:
      containers:
      - name: app
        image: my-app:2.0

Rolling Update (Default)

See Rolling Deployment section above.

Shadow Deployment

A copy of live traffic is redirected to the new version for testing without affecting production users.

- Most complex strategy
- Requires mock services
- Real traffic testing
- No user impact
- Downtime: No
- Use case: Performance validation

Choosing the Right Strategy

The right deployment strategy depends on your:

Risk tolerance
Budget constraints
Application architecture
Team capabilities
User base size

Decision Matrix

Strategy	Downtime	Complexity	Cost	Rollback Speed	Risk
Big Bang	High	Low	Low	Slow	High
Rolling	None	Medium	Low	Medium	Medium
Blue-Green	None	Medium	High	Instant	Low
Canary	None	High	Medium	Fast	Low
A/B Test	None	High	Medium	Fast	Low

Recommendations

Start Simple

Begin with rolling deployments for most applications

Add Safety

Implement canary for critical services

Invest in Monitoring

Good monitoring enables advanced strategies

Use Feature Flags

Decouple deployment from release

Reserve Blue-Green

Use for highest-risk deployments where instant rollback is critical

Best Practices Across All Strategies

1. Automate Everything

Use CI/CD pipelines
Automate testing
Automate rollbacks
Infrastructure as Code

2. Monitor Continuously

- Error rates
- Response times
- Resource usage
- Business metrics
- User experience

3. Test Thoroughly

Unit tests
Integration tests
Load tests
Smoke tests
Chaos engineering

4. Plan for Rollback

Always have a rollback plan
Test rollback procedures
Keep previous versions accessible
Document rollback steps

5. Communicate

Notify stakeholders
Maintain status pages
Document changes
Post-mortem after incidents

Successful deployment strategies combine technical implementation with good communication, monitoring, and process discipline.

APIs & Web

Databases

Caching & Performance

Security

Cloud & Distributed Systems

DevOps & CI/CD

​Why Deployment Strategies Matter

​Common Deployment Strategies

​Multi-Service (Big Bang) Deployment

​Overview

​Characteristics

​When to Use

​Rolling Deployment

​Overview

​Characteristics

​Configuration Example

​When to Use

​Blue-Green Deployment

​Overview

​Characteristics

​Implementation Example

​When to Use

​Canary Deployment

​Overview

​Characteristics

​Implementation Example

​Canary Metrics to Monitor

​When to Use

​A/B Testing

​Overview

​A/B Testing vs Canary

​Implementation Example

​Characteristics

​When to Use

​Feature Toggle (Feature Flags)

​Overview

​Types of Feature Flags

​Implementation Example

​Best Practices

​Kubernetes Deployment Strategies

​Recreate Strategy

​Rolling Update (Default)

​Shadow Deployment

​Choosing the Right Strategy

​Decision Matrix

​Recommendations

​Best Practices Across All Strategies

​1. Automate Everything

​2. Monitor Continuously

​3. Test Thoroughly

​4. Plan for Rollback

​5. Communicate

Build docs developers (and LLMs) love

Why Deployment Strategies Matter

Common Deployment Strategies

Multi-Service (Big Bang) Deployment

Overview

Characteristics

When to Use

Rolling Deployment

Overview

Characteristics

Configuration Example

When to Use

Blue-Green Deployment

Overview

Characteristics

Implementation Example

When to Use

Canary Deployment

Overview

Characteristics

Implementation Example

Canary Metrics to Monitor

When to Use

A/B Testing

Overview

A/B Testing vs Canary

Implementation Example

Characteristics

When to Use

Feature Toggle (Feature Flags)

Overview

Types of Feature Flags

Implementation Example

Best Practices

Kubernetes Deployment Strategies

Recreate Strategy

Rolling Update (Default)

Shadow Deployment

Choosing the Right Strategy

Decision Matrix

Recommendations

Best Practices Across All Strategies

1. Automate Everything

2. Monitor Continuously

3. Test Thoroughly

4. Plan for Rollback

5. Communicate