Working with Taints and Tolerations

1. Adding Taints to Nodes

Field	Description	Example
key	The key part of the taint	`node-role.kubernetes.io/master`
value	The value part of the taint	`reserved`
effect	The effect when pods do not tolerate the taint	`NoSchedule`, `PreferNoSchedule`, `NoExecute`

Example: Add a taint to a node

kubectl taint nodes <node-name> key=value:NoSchedule

Example: Add a taint with multiple keys and values

kubectl taint nodes <node-name> key1=value1:NoSchedule key2=value2:PreferNoSchedule

Note: The effect field determines how pods are handled when they don't tolerate the taint. NoSchedule prevents scheduling, PreferNoSchedule tries to avoid scheduling, and NoExecute evicts already running pods.

2. Removing Taints from Nodes

Command	Description	Notes
kubectl taint nodes <node-name> key<operator>	Remove a taint from a node	Use `key=` with `effect` to remove taints

Example: Remove taint from node

kubectl taint nodes worker-1 key1=effect1-

Note: The - at the end of the taint key is required to remove a taint.

3. Listing Node Taints

Example: List taints on a node

kubectl describe nodes <node-name> | grep -E '(Taint|Tolerations)'

Field	Description	Example
Taints	Lists taints applied to the node	`key1=value1:NoSchedule`
Tolerations	Lists tolerations that allow pods to be scheduled on the node	`key1=value1:NoSchedule`

Note: Taints are applied to nodes, while tolerations are applied to pods.

4. Understanding Taint Effects

Effect	Description	Behavior	Notes
NoSchedule	Prevents scheduling of pods that do not tolerate the taint	Pods will not be scheduled on the node unless they have a matching toleration	Most common effect for restricting pod placement
PreferNoSchedule	System tries to avoid scheduling pods on tainted nodes, but does not enforce it	Pods are scheduled on the node only if no other nodes are available or if they have a matching toleration	Acts as a soft constraint, not a hard rule
NoExecute	Evicts existing pods that do not tolerate the taint, and prevents new pods from being scheduled	Pods are evicted immediately if they don't tolerate the taint; new pods cannot be scheduled on the node	Used for maintenance or ensuring exclusive node usage

Example: Applying Taints with Different Effects

kubectl taint nodes node1 key1=value1:NoSchedule
kubectl taint nodes node1 key2=value2:PreferNoSchedule
kubectl taint nodes node1 key3=value3:NoExecute

Note: The NoExecute effect is particularly useful for node maintenance or ensuring dedicated resources for specific workloads.

5. Configuring Tolerations

Example: Pod with Tolerations

{
  "apiVersion": "v1",
  "kind": "Pod",
  "metadata": {
    "name": "example-pod"
  },
  "spec": {
    "tolerations": [
      {
        "key": "node-role.kubernetes.io/master",
        "effect": "NoSchedule"
      }
    ],
    "containers": [
      {
        "name": "example-container",
        "image": "nginx"
      }
    ]
  }
}

Field	Description	Values
key	The key of the taint to tolerate	`node-role.kubernetes.io/master`
effect	The effect of the taint to tolerate	`NoSchedule`, `PreferNoSchedule`, `NoExecute`
operator	Operator to use for matching taints	`Equal`, `Exists`
value	The value of the taint to tolerate (if operator is Equal)	Any string
tolerationSeconds	Duration to tolerate taint (if effect is NoExecute)	Integer in seconds

Note: Tolerations allow pods to be scheduled on nodes with matching taints. They do not guarantee scheduling, but make it possible.

6. Using Toleration Operators

Operator	Description	Example
Equal	Matches if the key and value are equal	`{ "key": "node-role.kubernetes.io/master", "operator": "Equal", "value": "true" }`
Exists	Matches if the key exists on the node, regardless of value	`{ "key": "node-role.kubernetes.io/master", "operator": "Exists" }`

Example: Using Equal Operator

tolerations:
- key: "node-role.kubernetes.io/master"
  operator: Equal
  value: "true"
  effect: NoSchedule

Example: Using Exists Operator

tolerations:
- key: "node-role.kubernetes.io/worker"
  operator: Exists
  effect: NoExecute

7. Setting Toleration Seconds

Field	Description	Values	Notes
tolerationSeconds	Duration in seconds for which the toleration is effective	`300`, `600`	Defaults to `null`, meaning the toleration is effective until the pod is scheduled or the node is removed

Example: Setting Toleration Seconds on a Pod

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  tolerations:
  - key: "key1"
    operator: "Equal"
    value: "value1"
    tolerationSeconds: 300
  containers:
  - name: example-container
    image: nginx

Note: The tolerationSeconds field is useful for pods that should be allowed to remain on a tainted node for a specific period, even if the taint is removed later.

8. Understanding Default Tolerations

Resource	Toleration	Effect	Notes
Node	`node.kubernetes.io/not-ready:NoExecute`	`NoExecute`	Prevents scheduling of pods on not-ready nodes
Node	`node.kubernetes.io/unreachable:NoExecute`	`NoExecute`	Prevents scheduling of pods on unreachable nodes
Pod	`kubernetes.io/cluster-autoscaler/uninitialized:NoSchedule`	`NoSchedule`	Allows cluster autoscaler to manage pods during initialization
Pod	`node.kubernetes.io/network-unavailable:NoSchedule`	`NoSchedule`	Prevents scheduling on nodes with unavailable network

Example: Default toleration in a Pod spec

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300
  - key: "node.kubernetes.io/unreachable"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 300

Note: These default tolerations are automatically added to pods when they are created, unless explicitly overridden in the pod spec.

9. Using Taints for Dedicated Nodes

Field	Description	Example
key	The key part of the taint	`dedicated`
value	The value part of the taint	`production`
effect	The effect when pods do not tolerate the taint	`NoSchedule`

Example: Taint a Node for Dedicated Use

kubectl taint nodes <node-name> dedicated=production:NoSchedule

Example: Pod with Tolerations for Dedicated Node

{
  "apiVersion": "v1",
  "kind": "Pod",
  "metadata": {
    "name": "prod-pod"
  },
  "spec": {
    "tolerations": [
      {
        "key": "dedicated",
        "operator": "Equal",
        "value": "production",
        "effect": "NoSchedule"
      }
    ],
    "containers": [
      {
        "name": "app",
        "image": "nginx"
      }
    ]
  }
}

Note: The NoSchedule effect prevents any pod without matching tolerations from being scheduled on the node.

10. Combining Taints with Node Affinity

Combining taints and node affinity allows for more granular control over pod scheduling by leveraging both node-level constraints (taints) and pod-level preferences (affinity).

Feature	Description	Usage
nodeTaints	Defines taints on nodes that repel pods without matching tolerations.	`key=value:effect`
nodeAffinity	Allows pods to prefer or require specific nodes based on labels.	`nodeAffinity.preferredDuringSchedulingIgnoredDuringExecution`
tolerations	Allows pods to tolerate specific taints and be scheduled on tainted nodes.	`effect:NoSchedule`

Example: Taints and Affinity Together

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
spec:
  tolerations:
  - key: "key1"
    operator: "Equal"
    value: "value1"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: "node-type"
            operator: "In"
            values: ["special"]

Note: Taints repel pods, while node affinity prefers or requires nodes. When used together, tolerations allow pods to bypass taints and still respect affinity rules.