Skip to main content
Mapagam

Debugging and Troubleshooting Pods

1. Viewing Pod Logs

Command Description Options
kubectl logs <pod-name> Retrieve logs from a pod -f to follow logs, --since=<duration> to filter by time
kubectl logs <pod-name> -c <container-name> Retrieve logs from a specific container in a pod -f, --since=<duration>
kubectl logs <pod-name> --previous Retrieve logs from a previous pod instance (e.g., after restart) -f, --since=<duration>

Example: Retrieve logs from a pod

kubectl logs my-pod

Example: Follow logs in real-time

kubectl logs -f my-pod

Example: Retrieve logs from a specific container

kubectl logs my-pod -c app-container
Note: If a pod has multiple containers, you must specify the container name to retrieve logs from that specific container.

2. Following Log Streams

Command Description Flags
kubectl logs Fetches logs from a pod -f to follow log output
kubectl logs -f Streams logs in real time -c <container> to specify container

Example: Follow logs from a pod

kubectl logs -f my-pod-name

Example: Follow logs from a specific container in a pod

kubectl logs -f my-pod-name -c app-container
Note: Use kubectl logs -f to continuously stream logs from a pod or container. This is useful for real-time debugging and monitoring.

3. Viewing Previous Container Logs

Command Description Notes
kubectl logs -p <pod-name> View logs from the previous instance of a container Useful for debugging issues after container restarts or crashes
kubectl logs --previous <pod-name> Alias for viewing previous container logs Same functionality as kubectl logs -p

Example: View Previous Logs

kubectl logs -p my-pod-1234567890-abcde
Note: Previous logs are only available if the container was restarted or replaced. If a pod is deleted and recreated, previous logs are lost.

4. Executing Commands in Pods

Command Description Example
kubectl exec Execute a command in a pod kubectl exec -it my-pod -- /bin/sh
kubectl exec -c Execute a command in a specific container within a pod kubectl exec -it my-pod -c sidecar-container -- /bin/bash
kubectl exec -i Execute a command in an interactive mode kubectl exec -it my-pod -i -- /bin/sh
kubectl exec -o Execute a command and output to file kubectl exec -it my-pod -- /bin/ls > output.txt

Example: Running a command in a pod

kubectl exec -it my-pod -- /bin/bash

Example: Running a command in a specific container

kubectl exec -it my-pod -c nginx -- /bin/sh

5. Creating Debug Containers

Property Description Example
debugContainer.image Image to use for the debug container busybox:latest
debugContainer.command Command to run in the debug container ["sleep", "3600"]
debugContainer.resources Resource limits and requests for the debug container 
{
  "limits": {
    "memory": "128Mi",
    "cpu": "100m"
  },
  "requests": {
    "memory": "64Mi",
    "cpu": "50m"
  }
}

Example: Creating a Debug Container

apiVersion: v1
kind: Pod
metadata:
  name: debug-pod
spec:
  containers:
  - name: main-container
    image: nginx:latest
  - name: debug-container
    image: busybox:latest
    command:
      - sleep
      - "3600"
    resources:
      limits:
        memory: "128Mi"
        cpu: "100m"
      requests:
        memory: "64Mi"
        cpu: "50m"
Note: Debug containers should not be used in production environments.

6. Using Ephemeral Containers

Ephemeral containers are a Kubernetes feature that allows you to add temporary containers to a running pod for debugging purposes without restarting the pod.

Property Description Values
name Name of the ephemeral container debugger
image Container image to use for debugging busybox:latest
command Command to run in the container ["sh"]
stdin Whether to attach stdin to the container true
tty Whether to allocate a TTY for the container true

Example: Adding an Ephemeral Container

apiVersion: v1
kind: Pod
metadata:
  name: debug-pod
spec:
  containers:
  - name: main-container
    image: nginx
  ephemeralContainers:
  - name: debugger
    image: busybox:latest
    command:
    - "sh"
    stdin: true
    tty: true
Note: Ephemeral containers are not persisted across pod restarts and are removed when the pod is deleted.

7. Viewing Pod Events

Event Type Description Example
Normal Informational events indicating successful operations Pulling image "nginx:latest"
Warning Events indicating potential issues or errors Back-off restarting failed container
Error Events indicating critical failures Failed to pull image "invalid-image:latest"

Example: Get Pod Events

kubectl get events --sort-by=.metadata.creationTimestamp

Example: Get Events for Specific Pod

kubectl get events --field-selector involvedObject.name=pod-name
Note: Events are time-stamped and sorted by creation timestamp. Use --sort-by to sort events.
Warning: Events are ephemeral and may be garbage collected after a period of time.

8. Using Port Forwarding

Port forwarding in Kubernetes allows you to access applications running inside pods from your local machine. This is particularly useful for debugging and development.

Command Description Example
kubectl port-forward Forward one or more local ports to a pod kubectl port-forward pod/my-pod 8080:80

Example: Forwarding a Local Port to a Pod

kubectl port-forward pod/nginx-deployment-7b4b5c9d8f-xyz12 8080:80
Note: The port-forwarding command will keep running until you terminate it with Ctrl+C.
Warning: Port forwarding is not suitable for production traffic and should only be used for debugging or development.

9. Inspecting API Objects

Resource Type Description Common Fields CLI Command
Pod Represents a set of running containers on a node metadata.name, spec.containers, status.phase kubectl get pod <name>
Service Exposes an application running in a pod metadata.name, spec.selector, spec.ports kubectl get svc <name>
Deployment Manages a set of replica pods metadata.name, spec.replicas, spec.template kubectl get deployment <name>
ConfigMap Stores configuration data as key-value pairs metadata.name, data kubectl get configmap <name>
Secret Stores sensitive data like passwords or tokens metadata.name, data kubectl get secret <name>

Example: Inspecting a Pod's Details

kubectl describe pod my-pod

Example: Getting Raw YAML of a Pod

kubectl get pod my-pod -o yaml
Note: Use kubectl describe for detailed event logs and status, while kubectl get -o yaml provides the full object definition.

10. Checking Cluster Events

Event Type Description Command Notes
Normal Standard operational events kubectl get events Default view, includes most pod and node events
Warning Non-critical issues that require attention kubectl get events --sort-by=.metadata.creationTimestamp Useful for identifying recent problems
Error Severe issues that may impact cluster stability kubectl get events -A | grep Error Filter by severity to focus on critical issues

Example: Viewing Events for a Specific Pod

kubectl get events --field-selector involvedObject.name=nginx-pod</code>
</pre>
</div>
</div>
<div class="example">
<h2 id="example-watching-events-in-real-time">Example: Watching Events in Real-Time<a href="#example-watching-events-in-real-time" class="toc__headingLink" aria-label="Link to this section"><svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg></a></h2>
<div class="codeBlock" data-language="bash">
<pre><code>kubectl get events --watch</code>
</pre>
</div>
</div>
<div class="note">
<strong>Note:</strong> Events are time-stamped and can help trace the sequence of actions leading up to a pod failure or unexpected behavior.
</div>
<h2 id="11-troubleshooting-networking">11. Troubleshooting Networking<a href="#11-troubleshooting-networking" class="toc__headingLink" aria-label="Link to this section"><svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg></a></h2>
<div class="example">
<h2 id="example-check-pod-network-connectivity">Example: Check Pod Network Connectivity<a href="#example-check-pod-network-connectivity" class="toc__headingLink" aria-label="Link to this section"><svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"></path><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"></path></svg></a></h2>
<div class="codeBlock" data-language="bash">
<pre><code>kubectl exec -it <pod-name> -- ping <other-pod-ip>
Issue Possible Cause Solution
Pod cannot reach other pods Network policies blocking traffic Check and update network policies to allow required traffic
Pod cannot resolve DNS DNS service not running or misconfigured Verify kube-dns or coredns is running and healthy
Pods in different namespaces cannot communicate Namespace isolation or service discovery issues Ensure services are properly exposed and use correct namespace references
Note: Use kubectl describe pod <pod-name> to check for network-related events or errors.

Example: Inspect Pod Network Interfaces

kubectl exec -it <pod-name> -- ip addr show

Workflow: Diagnose Pod Network Issues

  1. Check pod status and events using kubectl describe pod <pod-name>
  2. Verify network policies and namespace configurations
  3. Test connectivity between pods using kubectl exec
  4. Check DNS resolution inside the pod with kubectl exec -it <pod-name> -- nslookup
  5. Review logs from kube-dns or coredns if DNS issues persist

12. Using Node Debug Pods

Property Description Usage
debugger.image Container image used for debugging gcr.io/google-containers/debugger:latest
nodeSelector Selects nodes for debugging pod placement { "beta.kubernetes.io/os": "linux" }
tolerations Allows scheduling on tainted nodes [{ "effect": "NoSchedule", "key": "node-role.kubernetes.io/master" }]

Example: Create debug pod on node

apiVersion: v1
kind: Pod
metadata:
  name: debug-pod
spec:
  nodeSelector:
    beta.kubernetes.io/os: linux
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  containers:
  - name: debug-container
    image: gcr.io/google-containers/debugger:latest
    command:
      - /bin/bash
      - -c
      - echo "Debugging pod running on node"
Note: Ensure the node has sufficient resources to run the debug pod.
Warning: Debug pods should not be used in production environments.

Was this article helpful?

Related Cheat Sheets