Cluster Upgrades

​

Overview

​

Current Configuration

module "eks-production" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "18.4.0"
  cluster_name    = local.k8s_cluster_name
  cluster_version = "1.23"
  subnet_ids      = module.vpc.private_subnets
  vpc_id          = module.vpc.vpc_id
  eks_managed_node_groups = {
    spot = {
      desired_size = local.k8s_cluster_size
      max_size     = local.k8s_cluster_size
      min_size     = local.k8s_cluster_size

      create_launch_template = false
      launch_template_name   = ""
      disk_size              = 50
      instance_types         = ["r5d.xlarge"]
      capacity_type          = "SPOT"
    }
  }
  tags = {
    created-by = "terraform"
  }
}

• Current version: Kubernetes 1.23
• Terraform module: terraform-aws-modules/eks/aws v18.4.0
• Node type: Spot instances (r5d.xlarge)
• Capacity: Fixed size (desired = max = min)

​

Upgrade Planning

​

Version Support Policy

• Active support: Latest 3 minor versions
• Extended support: Additional 12 months (with fees)
• End of support: No security patches or support

​

Pre-Upgrade Checklist

1. Review Kubernetes changelog for breaking changes
2. Check API deprecations - Some APIs are removed in new versions
3. Test in staging environment if available
4. Backup cluster state - Export critical resources
5. Review addon compatibility - cert-manager, ingress controllers, etc.
6. Schedule maintenance window - Coordinate with team
7. Verify node capacity - Ensure enough nodes for workload during upgrade

​

Version Skipping

​

Control Plane Upgrade

​

Procedure

​

1. Update Terraform Configuration

module "eks-production" {
  ...
  cluster_version = "1.24"  # Increment by one minor version
  ...
}

​

2. Review Terraform Plan

cd terraform
terraform plan -target=module.eks-production

• Only cluster version is changing
• No node group recreations
• No unexpected resource deletions

​

3. Apply Upgrade

terraform apply -target=module.eks-production

• Control plane upgrade takes 20-45 minutes
• API server may have brief interruptions
• Running pods are not affected

​

4. Verify Control Plane

kubectl version --short
kubectl get nodes

​

Control Plane Upgrade Notes

• Zero downtime: Multiple API servers upgraded one at a time
• API compatibility: Old kubelet versions (N-2) still supported
• Automatic rollback: AWS automatically rolls back on failure
• No workload impact: Pods continue running during upgrade

​

Node Upgrade

​

Strategy

​

Option 1: Rolling Node Upgrade (Recommended)

​

1. Increase Node Group Capacity

eks_managed_node_groups = {
  spot = {
    desired_size = local.k8s_cluster_size * 2  # Double capacity
    max_size     = local.k8s_cluster_size * 2
    min_size     = local.k8s_cluster_size
    ...
  }
}

terraform apply -target=module.eks-production

​

2. Update Node AMI Version

eks_managed_node_groups = {
  spot = {
    ...
    # Add or update this to force node replacement
    update_config = {
      max_unavailable_percentage = 50
    }
  }
}

​

3. Apply Node Upgrade

terraform apply -target=module.eks-production

• Launch new nodes with updated AMI
• Wait for new nodes to be Ready
• Cordon old nodes
• Drain pods from old nodes
• Terminate old nodes

​

4. Monitor Node Replacement

# Watch node status
kubectl get nodes -w

# Check pod distribution
kubectl get pods -o wide --all-namespaces

​

5. Restore Original Capacity

eks_managed_node_groups = {
  spot = {
    desired_size = local.k8s_cluster_size  # Restore original size
    max_size     = local.k8s_cluster_size
    min_size     = local.k8s_cluster_size
    ...
  }
}

terraform apply -target=module.eks-production

​

Option 2: Blue/Green Node Group Upgrade

​

1. Create New Node Group

eks_managed_node_groups = {
  spot = { ... }  # Existing nodes
  
  spot-new = {
    desired_size = local.k8s_cluster_size
    max_size     = local.k8s_cluster_size
    min_size     = local.k8s_cluster_size
    disk_size    = 50
    instance_types = ["r5d.xlarge"]
    capacity_type  = "SPOT"
  }
}

​

2. Apply and Verify New Nodes

terraform apply -target=module.eks-production
kubectl get nodes

​

3. Drain Old Nodes

for node in $(kubectl get nodes -l eks.amazonaws.com/nodegroup=spot -o name); do
  kubectl drain $node --ignore-daemonsets --delete-emptydir-data --timeout=300s
done

​

4. Remove Old Node Group

eks_managed_node_groups = {
  spot-new = { ... }  # Rename to 'spot' if desired
}

terraform apply -target=module.eks-production

​

Post-Upgrade Verification

​

1. Verify Cluster Health

# Check node status
kubectl get nodes

# Verify all nodes are Ready and on new version
kubectl get nodes -o wide

# Check system pods
kubectl get pods -n kube-system

​

2. Verify Workloads

# Check all pods are running
kubectl get pods --all-namespaces | grep -v Running

# Verify services are accessible
kubectl get svc --all-namespaces

​

3. Test Application Functionality

• Run smoke tests on critical applications
• Verify ingress and load balancers
• Check certificate renewals
• Test database connections

​

4. Update Addons

# Check addon versions
aws eks describe-addon-versions --kubernetes-version 1.24

# Update kube-proxy
aws eks update-addon --cluster-name <cluster-name> --addon-name kube-proxy --addon-version <version>

# Update vpc-cni
aws eks update-addon --cluster-name <cluster-name> --addon-name vpc-cni --addon-version <version>

# Update coredns
aws eks update-addon --cluster-name <cluster-name> --addon-name coredns --addon-version <version>

​

Spot Instance Considerations

​

Spot Interruption Handling

resource "helm_release" "aws-node-termination-handler" {
  name       = "aws-node-termination-handler"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-node-termination-handler"
  version    = "0.13.0"
  namespace  = "kube-system"
  values     = [file("helm/aws-node-termination-handler.yaml")]
}

• Monitors for Spot interruption notices
• Cordons nodes before termination
• Drains pods gracefully (120s default)
• Prevents new pods from scheduling on interrupted nodes

​

Best Practices for Spot

1. Upgrade during low traffic - Reduces impact of spot interruptions
2. Monitor spot availability - Check instance availability in region
3. Use multiple instance types - Increase spot capacity pool
4. Set pod disruption budgets - Ensure minimum replicas during disruptions

​

Rollback Procedures

​

Control Plane Rollback

1. AWS automatic rollback - Control plane auto-reverts on failure
2. Manual intervention - Contact AWS Support if stuck

​

Node Rollback

​

Quick Rollback (if old nodes still exist)

1. Uncordon old nodes:
   kubectl uncordon <old-node-name>
   
2. Drain new nodes:
   kubectl drain <new-node-name> --ignore-daemonsets --delete-emptydir-data
   
3. Revert Terraform:
   git revert <commit-hash>
   terraform apply -target=module.eks-production
   

​

Troubleshooting

​

Nodes Not Joining Cluster

aws ssm start-session --target <instance-id>
sudo journalctl -u kubelet -n 100

• IAM role trust relationship incorrect
• Security group blocking kubelet communication
• aws-auth ConfigMap not updated

​

Pods Not Scheduling

kubectl describe node <node-name> | grep Taints

kubectl describe pod <pod-name>

​

API Server Unavailable

aws eks describe-cluster --name <cluster-name> --query 'cluster.status'

​

Best Practices

1. Stay current - Upgrade within 2 minor versions of latest
2. Test thoroughly - Use staging environment for validation
3. Upgrade incrementally - One minor version at a time
4. Schedule wisely - Low traffic periods, avoid Fridays
5. Monitor closely - Watch metrics during and after upgrade
6. Document changes - Note any configuration changes required
7. Communicate - Inform team of maintenance windows
8. Backup first - Export critical resources before upgrade
9. PodDisruptionBudgets - Ensure critical apps have PDBs
10. Addon compatibility - Verify all addons support new version

​

Emergency Contacts

• AWS Support: Use AWS Support Console for EKS issues
• On-call engineer: Check internal documentation
• Escalation: Follow incident response procedures