Managing Cluster Upgrades

1. Planning Cluster Upgrades

Upgrade Planning Steps

Assess current cluster version and compare with target version
Review release notes for breaking changes and new features
Check compatibility of applications and dependencies
Back up critical data and configurations
Test upgrade process in a staging environment
Plan maintenance windows and communication strategy
Validate resource requirements for new version

Version	Status	Notes
v1.28	NEW	Latest stable release with improved security features
v1.27	DEPRECATED	End of support as of 2024-06-01
v1.26	Supported	Current LTS version with security patches

Upgrade Strategies

Blue-Green Deployment: Deploy new cluster and switch traffic
Rolling Update: Gradually replace nodes in-place
Recreate: Fully recreate cluster with new version

Note: Always ensure etcd is backed up before any major upgrade to prevent data loss.

Warning: Some features may be deprecated or removed in newer versions. Review API deprecations carefully.

2. Upgrading Control Plane

Component	Version	Description	Status
kube-apiserver	`v1.28.0`	API server for Kubernetes cluster	NEW
etcd	`v3.5.7`	Highly available key-value store	DEPRECATED
kube-controller-manager	`v1.28.0`	Manages controller processes	NEW
kube-scheduler	`v1.28.0`	Schedules pods to nodes	NEW

Example: Upgrade Control Plane Components

sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.28.0

Note: Ensure all nodes are ready before upgrading control plane.

Warning: Back up etcd data before upgrading.

3. Upgrading Worker Nodes

Step	Description	Action
1	Drain the node to reschedule pods	`kubectl drain <node-name> --ignore-daemonsets`
2	Update the node's Kubernetes version	`kubectl uncordon <node-name>`
3	Verify the node is running the new version	`kubectl get nodes`

Note: Ensure that the node is cordoned off to prevent new pods from being scheduled on it during the upgrade process.

Warning: Draining a node may cause downtime for applications running on it. Ensure that your pods are configured with replicas to maintain availability.

Example: Drain and Upgrade Worker Node

kubectl drain worker-node-01 --ignore-daemonsets
kubectl uncordon worker-node-01

4. Upgrading kubectl

Version	Command	Notes
v1.29	`curl -LO "https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl"`	Latest stable release as of 2026
v1.28	`curl -LO "https://dl.k8s.io/release/v1.28.0/bin/linux/amd64/kubectl"`	Previous stable release

Example: Install kubectl v1.29

# Download kubectl v1.29
curl -LO "https://dl.k8s.io/release/v1.29.0/bin/linux/amd64/kubectl"

# Make it executable
chmod +x kubectl

# Move to PATH (e.g., /usr/local/bin)
sudo mv kubectl /usr/local/bin/

Note: Ensure the kubectl version is compatible with your cluster's API server version. A minor version skew of up to two versions is generally acceptable.

5. Upgrading Managed Clusters

Provider	Upgrade Method	Notes
Amazon EKS	`aws eks update-cluster-version`	Must specify a supported version; minor version upgrades only
Google GKE	`gcloud container clusters upgrade`	Supports automatic and manual upgrades; can set maintenance windows
Azure AKS	`az aks upgrade`	Can upgrade control plane and node pools separately; supports preview versions

Example: GKE Cluster Upgrade

gcloud container clusters upgrade my-cluster \
    --cluster-version=1.28.3-gke.1000 \
    --zone=us-central1-a

Note: Always ensure all workloads are compatible with the target Kubernetes version before upgrading.

Warning: Upgrading managed clusters can cause temporary downtime for workloads if not properly planned.

Upgrade Considerations

Backward compatibility of applications and CRDs
Scheduling and pod disruption budgets
Node image updates (if applicable)
Monitoring and alerting during upgrade

6. Using Blue-Green Strategy

Example: Deploying with Blue-Green Strategy

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: v1
  template:
    metadata:
      labels:
        app: myapp
        version: v1
    spec:
      containers:
      - name: app-container
        image: myapp:v1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: myapp
      version: v2
  template:
    metadata:
      labels:
        app: myapp
        version: v2
    spec:
      containers:
      - name: app-container
        image: myapp:v2
---
apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: myapp
    version: v1
  ports:
  - port: 80
    targetPort: 8080

Component	Description	Notes
Deployment (blue)	`app-blue`	Current production version
Deployment (green)	`app-green`	New version to deploy
Service Selector	`version: v1`	Routes traffic to blue deployment

Deployment Workflow

Create new app-green deployment with updated version
Verify green deployment is healthy
Update service selector from v1 to v2
Scale down blue deployment to zero replicas
Update service selector back to v2 if needed

Benefits of Blue-Green Deployment

Zero-downtime deployment
Easy rollback by switching traffic
Can test new version with real traffic
Reduces risk of deployment failure

Note: Ensure both deployments have identical resource requests and limits to avoid service disruption during switch.

Warning: Blue-green strategy requires double the resources (CPU, memory) for both versions during deployment.

7. Performing Rolling Upgrades

Step	Description	Action
1	Identify the target Kubernetes version	`kubectl version` to check current version, then decide on the target version.
2	Prepare the cluster for upgrade	Ensure all nodes are in a healthy state and have sufficient resources.
3	Update the control plane	`kubectl drain <node-name>` to safely evict pods and mark node as unschedulable.
4	Upgrade the control plane components	Use `kubeadm upgrade apply` to update the control plane.
5	Upgrade the worker nodes	Run `kubeadm upgrade node` on each worker node to update the kubelet and kubectl.
6	Verify the upgrade	Use `kubectl get nodes` to confirm all nodes are at the new version.

Example: Performing a Rolling Upgrade

kubectl drain node-1 --ignore-daemonsets
kubeadm upgrade apply v1.29.0
kubectl drain node-2 --ignore-daemonsets
kubeadm upgrade node
kubectl get nodes

Note: Always back up your cluster configuration and etcd before performing an upgrade.

Warning: Ensure that all applications are compatible with the new Kubernetes version before upgrading.

8. Testing Before Upgrade

Example: Pre-upgrade testing with kubectl

kubectl get pods --all-namespaces
kubectl get nodes
kubectl describe node <node-name>

Test Type	Description	Command	Status
Pod Health	Ensure all pods are running and healthy	`kubectl get pods --all-namespaces`	NEW
Node Status	Verify all nodes are in Ready state	`kubectl get nodes`	DEPRECATED
Node Details	Inspect node conditions and events	`kubectl describe node <node-name>`	BETA

Pre-Upgrade Checklist

Verify cluster health and stability
Ensure no ongoing deployments or updates
Backup critical configurations and data

Warning: Performing upgrades without pre-testing can lead to service disruptions and data loss.

Note: Use kubectl drain to safely evict pods from nodes before upgrade.

9. Rolling Back Upgrades

Step	Action	Description
1	`kubectl rollout undo`	Revert the deployment to the previous version
2	`kubectl rollout undo deployment/<deployment-name> --to-revision=<revision-number>`	Revert to a specific revision
3	`kubectl get deployments`	Verify the rollback was successful

Example: Rolling Back a Deployment

kubectl rollout undo deployment/myapp-deployment
kubectl rollout status deployment/myapp-deployment

Note: Rollback only works for deployments that support rolling updates. If the deployment strategy is not RollingUpdate, rollback may not be possible.

Warning: Ensure that the previous version is stable before performing a rollback to avoid introducing issues into the cluster.

10. Monitoring Upgrade Progress

Component	Metric	Description	Collection Method
Control Plane	`etcd_leader_changes`	Number of leader changes in etcd	Prometheus endpoint
Control Plane	`kubelet_node_status`	Node status reported by kubelet	Prometheus endpoint
Control Plane	`apiserver_request_total`	Total API server requests	Prometheus endpoint
Worker Nodes	`kube_node_status_condition`	Node status conditions (Ready, DiskPressure)	Prometheus endpoint
Worker Nodes	`kube_pod_status_phase`	Pod phase status (Running, Pending)	Prometheus endpoint

Example: Monitoring etcd leader changes

curl -H "Authorization: Bearer <token>" http://localhost:8080/metrics | grep etcd_leader_changes

Note: Use kubectl top to monitor resource usage during upgrade.

Warning: High etcd_leader_changes can indicate instability in the control plane.