Skip to main content
Mapagam

Implementing Horizontal Pod Autoscaling

1. Creating HPA

Property Description Values
apiVersion API version to use autoscaling/v2
kind Kind of resource HorizontalPodAutoscaler
metadata.name Name of the HPA resource my-hpa
spec.scaleTargetRef.apiVersion API version of the target resource apps/v1
spec.scaleTargetRef.kind Kind of the target resource Deployment
spec.scaleTargetRef.name Name of the target resource my-deployment
spec.minReplicas Minimum number of replicas 2
spec.maxReplicas Maximum number of replicas 10
spec.metrics Metrics to use for scaling Array of metric definitions

Example: HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
Note: The HPA will scale the target deployment based on CPU utilization, maintaining an average of 70%.

2. Creating HPA from YAML

Example: HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
Field Description Values
apiVersion API version of the HPA resource autoscaling/v2
kind Kind of the resource HorizontalPodAutoscaler
metadata.name Name of the HPA resource String
spec.scaleTargetRef Reference to the target resource to scale Object with apiVersion, kind, and name
spec.minReplicas Minimum number of replicas to maintain Integer
spec.maxReplicas Maximum number of replicas to scale up to Integer
spec.metrics Metrics to use for scaling decisions Array of metric objects
Note: The HPA will scale the target resource based on CPU utilization. Ensure that the target resource (e.g., Deployment) has resource requests defined for accurate scaling.

3. Listing HPAs

Command Description Notes
kubectl get hpa List all HPAs in the current namespace Default output includes name, reference, target, current, and age
kubectl get hpa -A List all HPAs across all namespaces Useful for cluster-wide HPA monitoring
kubectl get hpa -o wide List HPAs with additional details Shows CPU/Memory usage and target metrics
kubectl get hpa -o json Get HPA details in JSON format Useful for scripting and automation
kubectl get hpa -o yaml Get HPA details in YAML format Useful for configuration review and editing

Example: Listing HPAs in a Namespace

kubectl get hpa -n production
NAME         REFERENCE                    TARGETS         CURRENT   MINPODS   MAXPODS   REPLICAS   AGE
web-hpa      Deployment/web               70% / 80%       65%       2         10        5          2d
Note: The TARGETS column shows the current utilization percentage and the target threshold. If the current usage exceeds the target, the HPA will scale up the pods.

4. Describing HPA Details

Property Description Example
apiVersion API version of the HPA resource autoscaling/v2
kind Kind of the Kubernetes resource HorizontalPodAutoscaler
metadata.name Name of the HPA resource my-app-hpa
spec.scaleTargetRef Reference to the target resource (Deployment, StatefulSet, etc.) 
{
  "apiVersion": "apps/v1",
  "kind": "Deployment",
  "name": "my-app"
}
spec.minReplicas Minimum number of replicas allowed 2
spec.maxReplicas Maximum number of replicas allowed 10
spec.metrics Metrics used to determine scaling 
[
  {
    "type": "Resource",
    "resource": {
      "name": "cpu",
      "target": {
        "type": "Utilization",
        "averageUtilization": 70
      }
    }
  }
]

Example: Describe HPA in YAML

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
Note: HPA v2 supports multiple metric types including Resource, Pod, and External metrics.
Warning: Ensure that the target resource (e.g., Deployment) has a valid replicas field for HPA to scale it.

HPA Configuration Summary

  • apiVersion: Must be autoscaling/v2 for latest features
  • kind: Must be HorizontalPodAutoscaler
  • scaleTargetRef: Points to the resource to scale
  • minReplicas: Minimum pods allowed (default: 1)
  • maxReplicas: Maximum pods allowed
  • metrics: Defines how to scale (e.g., CPU, memory, custom)

5. Setting Min and Max Replicas

Field Description Values
minReplicas Minimum number of replicas that the HPA can scale down to 1
maxReplicas Maximum number of replicas that the HPA can scale up to 10

Example: HPA with min and max replicas

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: example-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-deployment
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

6. Using CPU-Based Scaling

Property Value Description
autoscaling/v2 HorizontalPodAutoscaler API version for HPA configuration
scaleTargetRef kind: Deployment<br>name: my-app Target resource to scale
minReplicas 2 Minimum number of pods to maintain
maxReplicas 10 Maximum number of pods to scale up to
metrics type: Resource<br>resource:<br>  name: cpu<br>  target:<br>    type: Utilization<br>    averageUtilization: 70 Target CPU utilization percentage

Example: HPA Configuration Using CPU Utilization

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: cpu-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
Note: HPA will scale based on average CPU usage across all pods. If a pod's CPU utilization exceeds 70%, new pods will be created until the average drops below the threshold.

7. Using Memory-Based Scaling

Property Value Description
memory 500Mi Memory threshold in Mebibytes (Mi) for scaling
memoryPercent 70% Percentage of total memory to use for scaling

Example: Memory-based HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: memory-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
Note: Memory-based autoscaling uses the average memory utilization across all pods to determine scaling actions. Ensure that your application's memory usage is consistent and predictable for accurate scaling.

8. Using Custom Metrics

Metric Type Description Usage Notes
external Custom metrics from external sources (e.g., Prometheus, Datadog) external in metrics section of HPA Requires metrics adapter (e.g., Prometheus Adapter)
object Metrics from a specific object (e.g., HTTP requests per second to a service) object in metrics section of HPA Uses target to define desired value per object
pods Metrics from all pods in the target resource (e.g., CPU usage per pod) pods in metrics section of HPA Uses target.averageValue or target.averageUtilization

Example: External Custom Metric

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: queue-length
          selector:
            matchLabels:
              queue: my-queue
        target:
          type: Value
          value: "10"

Example: Object Metric

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Object
      object:
        describedObject:
          apiVersion: v1
          kind: Service
          name: my-service
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"
Note: Custom metrics require a metrics adapter to expose the metric to the HPA controller. Prometheus Adapter is commonly used for this purpose.

9. Configuring Scale Up Behavior

Property Value Description
scaleTargetRef {apiVersion: "apps/v1", kind: "Deployment", name: "my-app"} Reference to the resource to scale
minReplicas 2 Minimum number of replicas to maintain
maxReplicas 10 Maximum number of replicas to scale up to
targetCPUUtilizationPercentage 70 Target average CPU usage percentage across all pods
behavior {scaleUp: {stabilizationWindowSeconds: 300, policies: [{type: "Percent", value: 100, periodSeconds: 60}]}, scaleDown: {stabilizationWindowSeconds: 300, policies: [{type: "Percent", value: 50, periodSeconds: 60}]}} Defines scale-up and scale-down behaviors

Example: HPA Configuration

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
Note: The stabilizationWindowSeconds setting prevents rapid scaling actions by waiting before applying new scale policies.
Warning: Setting maxReplicas too high can lead to resource exhaustion or cost overruns.

10. Configuring Scale Down Behavior

Property Description Default Value Notes
scaleDown.stabilizationWindowSeconds Number of seconds to wait before scaling down after a scale-up event 300 Used to prevent rapid scaling down after a scale-up, allowing the system to stabilize
scaleDown.policy Policy for scaling down pods default Can be set to default, all, or none
scaleDown.tolerance Threshold for scaling down pods 0.1 Percentage of CPU or memory utilization below which pods can be scaled down

Example: Configuring Scale Down Behavior in HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policy:
      - type: Percent
        value: 10
        periodSeconds: 60
Note: The scaleDown.stabilizationWindowSeconds setting helps prevent thrashing by ensuring that the system waits for a specified time before scaling down after a scale-up event.

11. Setting Stabilization Window

Property Description Default Value Notes
horizontalPodAutoscalerConfig.stabilizationWindowSeconds Defines the period in seconds over which the stabilization of metrics is considered. 300 Used to prevent rapid scaling actions based on short-term fluctuations in metrics.

Example: Configuring Stabilization Window in HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    stabilizationWindowSeconds: 300
Note: The stabilizationWindowSeconds is part of the HPA's behavior configuration and helps in avoiding flapping by smoothing out metric variations over a specified time window.

12. Troubleshooting HPA

Common HPA Issues and Solutions

  1. HPA Not Scaling
    • Check if the HPA is in a ScalingActive state using kubectl describe hpa <hpa-name>
    • Verify that the target resource (e.g., CPU or memory) is being monitored correctly
    • Ensure that the metrics server is running and accessible
  2. Scaling Too Frequently
    • Adjust scaleDownStabilizationWindow and scaleUpStabilizationWindow to prevent flapping
    • Review metric values and ensure they are not noisy or fluctuating rapidly
  3. HPA Not Responding to Metrics
    • Run kubectl top pods to confirm pod resource usage is reported
    • Check if metrics-server or prometheus-adapter is correctly configured
    • Ensure that the external or object metrics are correctly exposed and accessible
Issue Possible Cause Solution
HPA Not Scaling Metrics not available or incorrect target Verify metrics-server is running and HPA targets are correct
Scaling Too Frequently Fluctuating metrics or low thresholds Increase stabilization windows or adjust thresholds
HPA Not Responding to Metrics Missing or misconfigured metrics adapter Check prometheus-adapter or metrics-server configuration

Example: Debugging HPA Metrics

kubectl get hpa <hpa-name> -o yaml
kubectl describe hpa <hpa-name>
kubectl top pods
Note: Use kubectl top pods to verify that resource usage is being reported correctly by the metrics server.
Warning: If metrics-server is not running, HPA will not be able to scale based on CPU or memory usage.

Was this article helpful?

Related Cheat Sheets