Skip to main content
Mapagam

Implementing Cost Optimization

1. Right-Sizing Resources

Resource Type Property Recommended Value Notes
CPU Requests cpu 500m Start with 500m for most workloads
Memory Requests memory 512Mi Start with 512Mi for most workloads
CPU Limits cpu 1000m Set to 2x CPU requests for burstable workloads
Memory Limits memory 1Gi Set to 2x memory requests for burstable workloads

Example: Resource Configuration

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: example-container
    image: nginx
    resources:
      requests:
        memory: "512Mi"
        cpu: "500m"
      limits:
        memory: "1Gi"
        cpu: "1000m"
Note: Use kubectl top to monitor actual usage and adjust resources accordingly.
Warning: Setting limits too low can cause OOMKilled errors.

Right-Sizing Best Practices

  • Start with conservative resource requests (500m CPU, 512Mi memory)
  • Monitor actual usage over time
  • Increase limits to 2x the request values for burstable workloads
  • Use ResourceQuota to enforce limits at namespace level

2. Using Spot Instances

Field Description Values Notes
spotInstance Flag to enable spot instances true / false Default is false
maxPrice Maximum price to pay for spot instance $0.05 or $0.10 Set to prevent overpaying
instanceType Instance type to use for spot instances m5.large, c5.xlarge Must support spot instances

Example: Spot Instance Configuration

apiVersion: v1
kind: Pod
metadata:
  name: spot-pod
spec:
  containers:
  - name: app-container
    image: nginx
  nodeSelector:
    cloud.google.com/gke-spot: "true"
Note: Spot instances can be terminated at any time by the cloud provider if capacity is needed elsewhere.
Warning: Ensure your application can handle preemption of spot instances gracefully.
Provider Spot Instance Support Max Price Limit Notes
AWS EC2 NEW Yes Uses spot instance type
GCP Compute Engine NEW Yes Uses preemptible instances
Azure VMs NEW Yes Uses low priority VMs

3. Implementing Cluster Autoscaler

Configuration Parameter Description Default Value Notes
--node-group-auto-discovery Defines how to discover node groups for autoscaling auto Can be set to asg:<tag-key>=<tag-value>
--scale-down-delay-after-add Time to wait before scaling down after a scale-up event 10m Prevents rapid scale-downs after scale-ups
--scale-down-unneeded-time Time a node must be unneeded before it is considered for deletion 10m Helps avoid premature node deletion
--scale-down-gpu-unneeded-time Time a GPU node must be unneeded before deletion 10m Specific to GPU-enabled nodes
--max-node-provision-time Maximum time to wait for a node to be provisioned 15m Prevents indefinite waits for node creation

Example: Cluster Autoscaler Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-autoscaler
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cluster-autoscaler
  template:
    metadata:
      labels:
        app: cluster-autoscaler
    spec:
      containers:
      - image: k8s.gcr.io/autoscaling/cluster-autoscaler:v1.23.0
        name: cluster-autoscaler
        command:
          - /cluster-autoscaler
          - --v=4
          - --stderrthreshold=info
          - --cloud-provider=aws
          - --skip-nodes-with-local-storage=false
          - --expander=least-waste
          - --node-group-auto-discovery=asg:<tag-key>=<tag-value>
          - --scale-down-delay-after-add=10m
          - --scale-down-unneeded-time=10m
Note: Ensure that the IAM role or service account used by Cluster Autoscaler has appropriate permissions to manage EC2 instances or other cloud resources.
Warning: Misconfiguration of Cluster Autoscaler can lead to over-provisioning or under-provisioning, resulting in increased costs or performance issues.
Scaling Strategy Description Use Case
least-waste Chooses node groups that minimize waste of resources Optimizes resource usage and cost
most-pods Chooses node groups that can schedule the most pods Best for high pod density scenarios
priority Uses priority-based selection of node groups Allows custom scheduling logic

Example: Node Group Tagging for Discovery

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name my-asg \
  --launch-configuration-name my-lc \
  --min-size 1 \
  --max-size 10 \
  --desired-capacity 2 \
  --tags "Key=tag-key,Value=tag-value,PropagateAtLaunch=true"

4. Configuring Pod Disruption Budgets

A Pod Disruption Budget (PDB) ensures that applications remain available during voluntary disruptions like node maintenance or upgrades.

Field Description Example
metadata.name Name of the PDB object my-app-pdb
spec.minAvailable Minimum number of pods that must remain available during disruption 2
spec.maxUnavailable Maximum number of pods that can be unavailable during disruption 1
spec.selector.matchLabels Label selector to identify pods covered by this PDB {"app": "my-app"}

Example: Basic PDB Configuration

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app
Note: You must define either minAvailable or maxUnavailable, but not both.
Warning: If a PDB is too restrictive, it can prevent necessary disruptions like node upgrades.

Key Points

  • PDBs prevent voluntary disruptions from taking down too many pods at once.
  • Use minAvailable to ensure a minimum number of pods stay online.
  • Use maxUnavailable to allow a specific number of pods to be disrupted.
  • Pods must match the label selector in spec.selector.

5. Using Vertical Pod Autoscaler

The Vertical Pod Autoscaler (VPA) in Kubernetes automatically adjusts the CPU and memory requests and limits for pods based on their historical resource usage. It helps optimize resource allocation to reduce waste and improve cost efficiency.

Field Description Values
metadata.name Name of the VPA resource vpa-example
spec.targetRef Reference to the target resource (Deployment, StatefulSet, etc.) apiVersion: apps/v1<br>kind: Deployment<br>name: my-app
spec.updatePolicy.updateMode How the VPA updates pod resources Auto, Initial, Recreate

Example: VPA Configuration

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vpa-example
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
Note: VPA does not update resources for running pods. It requires a pod restart to apply changes.
Update Mode Description Behavior
Auto Automatically updates resources for new pods Applies recommendations to new pods
Initial Applies recommendations only to new pods Does not update existing pods
Recreate Recreates pods with updated resource limits Deletes and recreates pods to apply changes

Example: Resource Recommendation

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vpa-example
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: "app-container"
      minAllowed:
        cpu: "200m"
        memory: "128Mi"
      maxAllowed:
        cpu: "2"
        memory: "1Gi"

6. Scheduling Workloads Efficiently

Field Description Values Notes
nodeSelector Map of node labels to match { "kubernetes.io/os": "linux" } Basic label-based scheduling
affinity Advanced scheduling rules { nodeAffinity: { requiredDuringSchedulingIgnoredDuringExecution: { ... } } } More flexible than nodeSelector
tolerations Allow pods to be scheduled on tainted nodes { key: "node-role.kubernetes.io/master", operator: "Equal", value: "true", effect: "NoSchedule" } Used to override taints
priorityClassName Priority class name to set scheduling priority high-priority Higher priority pods may preempt lower ones

Example: Node Affinity Configuration

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: kubernetes.io/e2e-features
                operator: In
                values:
                  - network-policy
  containers:
    - name: nginx
      image: nginx:latest

Example: Tolerations for Master Node

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-toleration
spec:
  tolerations:
    - key: "node-role.kubernetes.io/master"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
  containers:
    - name: app
      image: myapp:latest

Key Concepts

  • nodeSelector: Simple label matching for basic scheduling.
  • affinity: More powerful than nodeSelector, supports soft and hard rules.
  • tolerations: Allow pods to run on nodes with taints.
  • priorityClassName: Influences scheduling order based on priority levels.

7. Implementing Resource Quotas

Resource Description Usage
limits.cpu Maximum CPU allocation for a pod 2 or 2000m
limits.memory Maximum memory allocation for a pod 512Mi or 1Gi
requests.cpu Minimum CPU guaranteed to a pod 500m
requests.memory Minimum memory guaranteed to a pod 256Mi

Example: Setting Resource Limits in a Pod

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: app-container
    image: nginx
    resources:
      limits:
        memory: "512Mi"
        cpu: "500m"
      requests:
        memory: "256Mi"
        cpu: "250m"

Example: Creating a ResourceQuota in a Namespace

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
Note: ResourceQuotas apply to the namespace and enforce limits on total resource consumption across all pods in that namespace.

8. Using Storage Classes Wisely

Property Description Default Value Notes
allowVolumeExpansion Enables volume expansion for volumes created from this StorageClass false Set to true to allow resizing of volumes
volumeBindingMode Determines when volume binding and dynamic provisioning occurs Immediate Can be Immediate or WaitForFirstConsumer
reclaimPolicy Determines what happens to the volume after the PVC is deleted Retain Can be Retain, Recycle, or Delete

Example: Creating a StorageClass with optimized settings

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: optimized-ssd
provisioner: aws-ebs
parameters:
  type: gp2
  fsType: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
Note: Using WaitForFirstConsumer can reduce costs by avoiding unnecessary volume provisioning.
Warning: Setting reclaimPolicy to Delete will permanently delete volumes when PVCs are removed.
Storage Class Volume Binding Mode Reclaim Policy Cost Optimization Benefit
fast-ssd Immediate Delete High performance, low cost for ephemeral workloads
slow-hdd WaitForFirstConsumer Retain Cost savings for long-term storage

9. Cleaning Up Unused Resources

Resource Type Cleaning Method Notes
Deployments kubectl delete deployment <name> Removes the deployment and its associated pods
Services kubectl delete service <name> Deletes the service and stops routing traffic to it
ConfigMaps kubectl delete configmap <name> Removes configuration data used by pods
Secrets kubectl delete secret <name> Deletes sensitive data like passwords or tokens
PersistentVolumes kubectl delete pv <name> Releases storage resources that are no longer in use
Ingress kubectl delete ingress <name> Removes routing rules for external traffic

Example: Delete Unused Deployment

kubectl delete deployment my-app-deployment

Example: Delete Unused Service

kubectl delete service my-app-service
Note: Always verify that the resource is no longer needed before deletion to avoid unintended downtime or data loss.

10. Monitoring Cost Allocation

Resource Cost Metric Description Monitoring Tool
CPU Usage millicores Amount of CPU consumed by pods Metrics Server
Memory Usage megabytes Amount of RAM consumed by pods Metrics Server
Storage Usage gigabytes Disk space used by persistent volumes Storage Class
Network I/O megabytes/second Data transfer rate in/out of pods CoreDNS

Example: Resource Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota
spec:
  hard:
    cpu: "1"
    memory: 1Gi
    persistentvolumeclaims: "1"
    requests.storage: 5Gi
Note: Resource quotas help control resource consumption and prevent over-allocation.
Warning: Setting quotas too low can cause application failures.

Key Points

  • Monitor resource usage to identify cost drivers.
  • Use Metrics Server for CPU and memory metrics.
  • Implement ResourceQuota to limit resource consumption.

Was this article helpful?

Related Cheat Sheets