Skip to main content
Mapagam

Optimizing Cluster Performance

1. Optimizing API Server

Configuration Option Description Default Value Notes
--etcd-servers Comma-separated list of etcd servers http://localhost:2379 Must be accessible to API server
--audit-log-path Path to audit log file None Enables audit logging for security monitoring
--enable-admission-plugins Comma-separated list of admission control plugins AlwaysAdmit Controls resource creation and modification
--max-requests-inflight Maximum inflight requests to API server 400 Adjust based on cluster size and load
--max-mutating-requests-inflight Maximum inflight mutating requests 200 Separate limit for write operations
--request-timeout Timeout for API requests 60s Prevents long-running requests from blocking

Example: API Server Configuration

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  containers:
  - name: kube-apiserver
    image: k8s.gcr.io/kube-apiserver:v1.29.0
    command:
    - kube-apiserver
    - --etcd-servers=http://etcd-server:2379
    - --audit-log-path=/var/log/audit.log
    - --enable-admission-plugins=NamespaceLifecycle,LimitRanger
    - --max-requests-inflight=500
    - --request-timeout=90s
Note: Adjust --max-requests-inflight and --max-mutating-requests-inflight based on your cluster's expected load to prevent overwhelming the API server.
Warning: Setting --request-timeout too low can cause timeouts for legitimate long-running operations.

2. Tuning etcd Performance

Parameter Description Default Value Notes
etcd-quota-backend-bytes Maximum backend size in bytes 8589934592 Set to 8GB for large clusters
etcd-compaction-mode Compaction mode for etcd periodic Use periodic for performance
etcd-compact-interval Interval between compactions 5m0s Adjust based on data growth rate
etcd-heartbeat-interval Heartbeat interval for etcd members 100ms Reduce for low-latency networks
etcd-election-timeout Timeout for leader election 1000ms Should be 10x heartbeat interval

Example: etcd Configuration in kube-apiserver

- --etcd-quota-backend-bytes=10737418240
- --etcd-compaction-mode=periodic
- --etcd-compact-interval=10m0s
- --etcd-heartbeat-interval=200ms
- --etcd-election-timeout=2000ms
Note: Increasing etcd-quota-backend-bytes can improve performance for large clusters but increases memory usage.
Warning: Setting etcd-election-timeout too low can cause frequent leader elections.

3. Optimizing Node Resources

Resource Type Configuration Description
cpu "cpu": "500m" Amount of CPU to allocate to a container. 'm' denotes milli-cores (e.g., 500m = 0.5 CPU core)
memory "memory": "512Mi" Amount of memory to allocate. 'Mi' = Mebibytes (1024^2 bytes)
ephemeral-storage "ephemeral-storage": "2Gi" Amount of ephemeral storage to allocate. 'Gi' = Gibibytes (1024^3 bytes)

Example: Resource Requests and Limits in a Pod Spec

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  containers:
  - name: app-container
    image: nginx
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
Note: Setting requests ensures the container gets scheduled on a node with enough resources. limits prevent containers from consuming more than their allocated resources.
Resource Default Value Impact of Over-allocation
cpu 100m (0.1 CPU core) Pods may be throttled or starved if over-requested
memory 128Mi Pods may be evicted if over-requested and node runs out of memory
ephemeral-storage 2Gi Pods may be evicted if storage limits are exceeded

4. Configuring DNS Optimization

Configuration Description Default Value Notes
coredns CoreDNS is the default DNS server in Kubernetes Enabled Used for service discovery and DNS resolution
clusterDomain Domain used for DNS resolution in the cluster cluster.local Customizable to match organizational standards
resolvConf Path to the resolv.conf file on nodes /etc/resolv.conf Used to configure upstream DNS servers

Example: Configuring CoreDNS with Custom Upstream Servers

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 1.1.1.1
        cache 30
        loop
        reload
        loadbalance
    }
Note: The forward directive in CoreDNS can be used to route queries to upstream DNS servers like Google Public DNS (8.8.8.8) or Cloudflare (1.1.1.1).

Example: Enabling Cache in CoreDNS

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          upstream
          fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . 8.8.8.8 1.1.1.1
        cache 30
        loop
        reload
        loadbalance
    }
Warning: Caching can lead to stale DNS records in highly dynamic environments. Ensure cache TTLs are tuned appropriately.

5. Using Node Local DNS Cache

Configuration Description Default Value Notes
nodeLocalDNS.enabled Enables NodeLocal DNS cache false Set to true to enable
nodeLocalDNS.port Port to listen on for DNS queries 53 Must be > 1024 for non-root users
nodeLocalDNS.upstreamNameservers Upstream DNS servers to forward queries to ["8.8.8.8", "8.8.4.4"] Can be customized for private DNS

Example: Enabling NodeLocal DNS in Helm Values

nodeLocalDNS:
  enabled: true
  port: 53
  upstreamNameservers:
    - "8.8.8.8"
    - "8.8.4.4"
Note: NodeLocal DNS improves DNS resolution latency by caching queries locally on each node.
Warning: Ensure that the nodeLocalDNS.port is not blocked by firewall rules or other network policies.
Feature Status Version Introduced Notes
NodeLocal DNS NEW Kubernetes 1.15+ Improves DNS performance and reduces load on CoreDNS

6. Reducing Image Pull Time

Option Values Notes
imagePullPolicy Always, IfNotPresent, Never IfNotPresent avoids re-pulling if image exists locally
imagePullSecrets secretName Use to authenticate with private registries
imageRepository registry.example.com/myapp:v1.2.3 Use regional or edge-cached registries for faster pulls

Example: Optimizing Image Pull Policy

apiVersion: v1
kind: Pod
metadata:
  name: optimized-pod
spec:
  containers:
  - name: app-container
    image: myapp:latest
    imagePullPolicy: IfNotPresent

Example: Using Image Pull Secrets

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  imagePullSecrets:
    - name: my-registry-secret
  containers:
  - name: app-container
    image: private.registry.com/myapp:v1.2.3
Note: Pre-pulling images using DaemonSets or init containers can reduce pull times in large clusters.

7. Optimizing Pod Startup

Field Description Example
imagePullSecrets Secrets to use for pulling images from private registries [{name: "my-secret"}]
imagePullPolicy When to pull images (Always, Never, IfNotPresent) IfNotPresent
restartPolicy Policy for restarting containers (Always, OnFailure, Never) Always
terminationGracePeriodSeconds Time to wait before killing a container after SIGTERM 30

Example: Pod with optimized startup settings

apiVersion: v1
kind: Pod
metadata:
  name: optimized-pod
spec:
  containers:
  - name: app-container
    image: myapp:latest
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 8080
  imagePullSecrets:
  - name: registry-secret
  restartPolicy: Always
Note: Setting imagePullPolicy to IfNotPresent can reduce startup time by avoiding unnecessary image pulls.
Warning: Setting terminationGracePeriodSeconds too low may cause containers to be killed before graceful shutdown.
Optimization Description Impact
Pre-pull images Use imagePullPolicy: IfNotPresent with pre-pulled images Reduces startup delay
Reduce container count Combine multiple services into single containers Decreases startup overhead
Use readiness probes Ensure containers are ready before traffic is routed Improves reliability

8. Using Pod Priority

Property Description Value
priorityClassName Defines the priority class for a Pod high-priority
priority Numeric value defining priority level (higher = more important) 1000000

Example: Setting Pod Priority

apiVersion: v1
kind: Pod
metadata:
  name: priority-pod
spec:
  priorityClassName: high-priority
  containers:
  - name: app
    image: nginx
Note: Priority classes must be defined before pods can use them.

9. Right-Sizing Resources

Resource Type Property Recommended Value Notes
CPU Requests cpu 500m Start with 500m for most workloads
Memory Requests memory 512Mi Start with 512Mi for most workloads
CPU Limits cpu 1000m Set to 2x CPU requests for burstable workloads
Memory Limits memory 1Gi Set to 2x memory requests for burstable workloads

Example: Right-sizing a Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: right-sized-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: right-sized-app
  template:
    metadata:
      labels:
        app: right-sized-app
    spec:
      containers:
      - name: app-container
        image: myapp:latest
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
Note: Use kubectl top to monitor actual resource usage and adjust accordingly.
Warning: Setting limits too low can cause OOMKilled errors.

10. Monitoring Cluster Performance

Metric Description Collection Method Recommended Tools
CPU Usage Percentage of CPU utilized by pods and nodes Metrics API, cAdvisor Prometheus, Grafana
Memory Usage Amount of RAM consumed by containers and nodes Metrics API, cAdvisor Prometheus, Grafana
Network I/O Bytes sent and received by pods Metrics API, cAdvisor Prometheus, Grafana
Disk I/O Read/write operations on storage devices Metrics API, cAdvisor Prometheus, Grafana
Pod Restart Count Number of times a pod has restarted Pod status, Events API Kubernetes Dashboard, Prometheus
Node Status Readiness and condition of nodes Node status, Events API Kubernetes Dashboard, Prometheus

Example: Querying CPU Usage with Prometheus

rate(container_cpu_usage_seconds_total{container!=""}[5m])

Example: Monitoring Memory Usage in Grafana

query: |
  sum(container_memory_usage_bytes{container!=""}) by (pod)
Note: Ensure that Prometheus is properly configured to scrape metrics from the Kubernetes API server and nodes.
Warning: Frequent pod restarts may indicate resource constraints or application issues.

Was this article helpful?

Related Cheat Sheets