Analyzing Resource Usage

1. Installing Metrics Server

Example: Install Metrics Server via kubectl

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Component	Description	Notes
metrics-server	`metrics-server`	Collects resource usage data from nodes and pods
API Group	`metrics.k8s.io`	Used to query resource metrics

Note: Ensure metrics-server is installed in the kube-system namespace.

Warning: The metrics-server requires RBAC permissions to access node and pod metrics.

2. Viewing Node Metrics

Metric	Description	Collection Method	Notes
cpu.usage	Current CPU usage percentage	`metrics.k8s.io`	Requires Metrics Server
memory.usage	Current memory usage in bytes	`metrics.k8s.io`	Requires Metrics Server
network.rx	Received network traffic in bytes	`metrics.k8s.io`	Requires CNI plugin support
network.tx	Transmitted network traffic in bytes	`metrics.k8s.io`	Requires CNI plugin support

Example: Fetching Node Metrics via kubectl

kubectl top node

Note: Ensure the Metrics Server is installed and running to collect node metrics.

3. Viewing Pod Metrics

Metric	Description	Collection Method	Notes
cpu	CPU usage in cores	`metrics.k8s.io` API	Requires Metrics Server
memory	Memory usage in bytes	`metrics.k8s.io` API	Requires Metrics Server
network	Network I/O statistics	`metrics.k8s.io` API	Not available in default Metrics Server

Example: Get Pod CPU and Memory Usage

kubectl top pod <pod-name>

Example: Get All Pods' Metrics

kubectl top pods

Note: The kubectl top command requires the Metrics Server to be installed in the cluster.

Warning: Metrics Server may not provide real-time data; it aggregates metrics periodically.

4. Viewing Container Metrics

Metric	Description	Collection Method	Notes
CPU Usage	Percentage of CPU cycles consumed by the container	`kubectl top pod <pod-name>`	Requires metrics-server to be installed
Memory Usage	Amount of RAM used by the container	`kubectl top pod <pod-name>`	Shows current usage and limits
Network I/O	Bytes sent and received by the container	Not directly available via kubectl; requires Prometheus or similar	Use `cadvisor` for detailed network stats
Disk I/O	Read/write operations on disk by container	Not directly available via kubectl; requires Prometheus or similar	Use `cadvisor` for detailed disk stats

Example: Fetching Pod Metrics

kubectl top pod my-app-pod

Note: The kubectl top command relies on the metrics-server, which must be installed and running in your cluster to provide real-time container resource usage data.

5. Sorting by Resource Usage

Field	Description	Usage
usage.cpu	CPU usage in cores	Sort by descending to identify high CPU consumers
usage.memory	Memory usage in bytes	Sort by descending to identify high memory consumers
usage.pods	Number of pods using resources	Sort by descending to identify high pod count consumers

Example: Sort pods by CPU usage

kubectl top pods --sort-by=cpu

Example: Sort nodes by memory usage

kubectl top nodes --sort-by=memory

6. Using Prometheus

Example: Querying CPU Usage

rate(container_cpu_usage_seconds_total{container!=""}[5m])

Query	Description	Notes
container_cpu_usage_seconds_total	Total CPU time used by containers	Rate over 5 minutes
container_memory_usage_bytes	Memory usage in bytes	Per container
container_network_receive_bytes_total	Total bytes received by container network	Per container

Note: Prometheus metrics are exposed via the /metrics endpoint on each pod or service.

Example: Setting up a Prometheus Rule

groups:
- name: kubernetes
  rules:
  - alert: HighCPUUsage
    expr: rate(container_cpu_usage_seconds_total{container!=""}[5m]) > 0.8
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"

7. Creating Grafana Dashboards

Example: Basic Kubernetes Metrics Dashboard

{
  "dashboard": {
    "title": "Kubernetes Cluster Metrics",
    "panels": [
      {
        "title": "CPU Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(container_cpu_usage_seconds_total{image!&#x3d;&#x27;&#x27;,container!&#x3d;&#x27;POD&#x27;}[5m])"
          }
        ]
      },
      {
        "title": "Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "container_memory_usage_bytes{image!&#x3d;&#x27;&#x27;,container!&#x3d;&#x27;POD&#x27;}"
          }
        ]
      }
    ]
  }
}

Panel Type	Description	Use Case
graph	Line graphs for time-series data	Monitor resource trends over time
stat	Single value display with trend	Show current CPU or memory usage
table	Tabular data display	Display pod or node resource allocation

Note: Ensure Prometheus is properly configured to scrape Kubernetes metrics for accurate dashboard data.

Warning: Grafana dashboards should be secured with authentication to prevent unauthorized access to cluster metrics.

8. Setting Up Alerts

Example: Prometheus Rule for CPU Usage

groups:
- name: cpu-alerts
  rules:
  - alert: HighCPUUsage
    expr: rate(container_cpu_usage_seconds_total{container!=""}[5m]) > 0.8
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"
      description: "Container {{ $labels.container }} on {{ $labels.instance }} is using more than 80% CPU for 10 minutes."

Alert Rule Field	Description	Example Value
alert	Name of the alert	`HighCPUUsage`
expr	Prometheus query expression to evaluate	`rate(container_cpu_usage_seconds_total{container!=""}[5m]) > 0.8`
for	Duration to wait before firing the alert	`10m`
labels	Additional metadata for the alert	`{ severity: warning }`
annotations	Human-readable descriptions and summaries	`{ summary: "High CPU usage", description: "Container ... is using more than 80% CPU..." }`

Note: Alerts are evaluated based on the Prometheus query language (PromQL). The for field allows for alert deduplication and ensures that alerts are only fired after a condition has persisted for a specified duration.

Example: Alertmanager Configuration

route:
  group_by: [alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: slack-notifications

receivers:
- name: slack-notifications
  slack_configs:
  - channel: '#alerts'
    send_resolved: true
    text: '<div class="alert-text">{{ .CommonAnnotations.summary }}</div>'

Alerting Workflow Overview

Define alert rules in Prometheus
Configure Alertmanager to handle alerts
Set up notification channels (Slack, Email, etc.)
Monitor and adjust thresholds as needed

9. Using kube-state-metrics

Example: Collecting Pod Metrics

apiVersion: v1
kind: Service
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  ports:
  - port: 8080
    name: http-metrics
  selector:
    app: kube-state-metrics

Resource	Description	Metrics Exposed
kube_pod_info	Information about pods including namespace, node, and labels	`pod`, `namespace`, `node`
kube_pod_status_phase	Current phase of the pod (Pending, Running, Succeeded, Failed)	`phase`
kube_pod_container_status_ready	Readiness status of containers within a pod	`ready`
kube_pod_container_resource_requests	Resource requests for containers in a pod	`cpu`, `memory`
kube_pod_container_resource_limits	Resource limits for containers in a pod	`cpu`, `memory`

Note: kube-state-metrics does not collect actual runtime metrics like CPU or memory usage; it only exposes metadata and state information.

10. Monitoring Control Plane

Component	Metric	Description	Collection Method
API Server	`apiserver_request_total`	Total number of API requests	Prometheus endpoint
Scheduler	`scheduler_queue_length`	Length of the scheduling queue	Prometheus endpoint
Controller Manager	`controller_manager_reconcile_duration_seconds`	Duration of reconciliation loops	Prometheus endpoint
etcd	`etcd_server_proposals_failed_total`	Number of failed proposals	Prometheus endpoint

Example: Collecting API Server Metrics

# Query API server metrics using kubectl
kubectl get --raw "/metrics" | grep apiserver_request_total

Note: Ensure Prometheus is configured to scrape control plane components for effective monitoring.