Skip to main content
Mapagam

Implementing Backup and Disaster Recovery

1. Installing Velero

Velero is a tool for backing up and restoring Kubernetes cluster resources and persistent volumes. It supports both on-premises and cloud environments.

Component Description Version
Velero Server Core component that runs in the Kubernetes cluster to manage backups and restores. 1.12.0
Velero CLI Command-line interface to interact with Velero server. 1.12.0
Backup Storage Provider Storage backend for storing backups (e.g., AWS S3, GCP Bucket, Azure Blob). Any supported cloud or local storage

Example: Installing Velero CLI

# Install Velero CLI using Homebrew (macOS)
brew install velero

# Or download binary directly
curl -LO https://github.com/vmware-tanzu/velero/releases/download/v1.12.0/velero-v1.12.0-linux-amd64.tar.gz
tar -xzf velero-v1.12.0-linux-amd64.tar.gz
sudo mv velero-v1.12.0-linux-amd64/velero /usr/local/bin/

Example: Installing Velero Server with AWS

# Set environment variables
export AWS_ACCESS_KEY_ID=your-access-key
export AWS_SECRET_ACCESS_KEY=your-secret-key
export REGION=us-west-2

# Install Velero with AWS plugin
velero install \
  --provider aws \
  --bucket velero-backups \
  --secret-name velero-aws \
  --use-volume-snapshots=true \
  --backup-location-config region=$REGION \
  --snapshot-location-config region=$REGION
Note: Ensure that the IAM role or user has appropriate permissions to access the backup storage and manage EC2 snapshots if using volume snapshots.

2. Configuring Backup Storage

Storage Type Description Use Case Notes
Local Storage Uses local disk or mounted storage on the Kubernetes nodes. Development/testing environments Not recommended for production due to lack of persistence and redundancy.
Network Attached Storage (NAS) Storage accessible over a network, such as NFS. Shared storage for multiple pods Requires network configuration and access control.
Cloud Storage (AWS S3, GCP Cloud Storage) Object storage services provided by cloud providers. Production-grade backups with high availability Requires IAM roles and permissions for access.
Persistent Volume (PV) with CSI Driver Uses Container Storage Interface (CSI) to integrate external storage systems. Dynamic provisioning and management of storage Supports various storage backends including local, cloud, and enterprise storage.

Example: Configuring a Persistent Volume for Backup Storage

apiVersion: v1
kind: PersistentVolume
metadata:
  name: backup-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  nfs:
    server: nfs-server.example.com
    path: /backup

Example: Using Cloud Storage with AWS S3

apiVersion: v1
kind: Secret
metadata:
  name: aws-credentials
type: Opaque
data:
  AWS_ACCESS_KEY_ID: <base64-encoded-access-key>
  AWS_SECRET_ACCESS_KEY: <base64-encoded-secret-key>

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: backup-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

3. Creating Backups

Backup Type Description Use Case Tools
etcd Backup Backs up the Kubernetes control plane state Disaster recovery for cluster state kube-apiserver, etcd
Persistent Volume Backup Backs up data stored in PersistentVolumes Data persistence across cluster failures Velero, Restic
ConfigMap/Secret Backup Backs up configuration and sensitive data Recovery of application settings kubectl get, Velero

Example: Backup etcd

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cert=/etc/etcd/pki/etcd.pem \
  --key=/etc/etcd/pki/etcd-key.pem \
  --cacert=/etc/etcd/pki/ca.pem \
  snapshot save /backup/etcd-snapshot.db

Example: Backup ConfigMaps and Secrets

kubectl get configmaps -o yaml --all-namespaces > configmaps.yaml
kubectl get secrets -o yaml --all-namespaces > secrets.yaml
Note: Ensure backups are encrypted and stored securely.
Warning: etcd snapshots must be taken while the cluster is healthy.

4. Scheduling Backups

Field Description Example
schedule Cron expression for backup frequency 0 0 * * *
template Backup template to use backup-template.yaml
retention Number of backups to keep 7

Example: Cron Schedule

schedule: 0 0 * * *

Example: Backup Template

template:
  metadata:
    name: backup-template
  spec:
    target:
      namespace: myapp
Note: Ensure the cron schedule aligns with your compliance requirements.

5. Restoring from Backup

Example: Restore a Kubernetes StatefulSet from Backup

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: restored-statefulset
spec:
  serviceName: "web"
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx:1.19
        ports:
        - containerPort: 80
        volumeMounts:
        - name: web-data
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: web-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi
Field Description Notes
apiVersion API version of the StatefulSet resource Must be apps/v1
kind Resource type Must be StatefulSet
metadata.name Name of the restored StatefulSet Must match backup name or be unique
spec.serviceName Name of the headless service Must match original service name
spec.replicas Number of replicas to restore Should match original replica count
spec.template.spec.containers[].image Container image to use for restored pods Must be same or compatible version
spec.volumeClaimTemplates[].spec.accessModes Access mode for persistent volumes Must match original access mode
Note: Ensure that the storage class and volume size match those used in the original backup.
Warning: Restoring a StatefulSet may cause data inconsistency if the backup was not taken at a consistent point in time.

Workflow: Restore Process Steps

  1. Verify backup exists and is accessible
  2. Identify the correct namespace for restoration
  3. Apply the restored StatefulSet manifest using kubectl apply -f
  4. Monitor pod status and logs for any issues
  5. Validate data integrity post-restoration

Summary of Restoration Considerations

  • Ensure backup was created with kubectl get or equivalent tooling
  • Validate that all dependencies (e.g., services, secrets) are present
  • Check for any namespace-specific configurations that may differ
  • Confirm that restored resources have the same labels and selectors as original

6. Backing Up Specific Namespaces

Namespace Description Backup Method Notes
frontend-app Contains web application pods and services kubectl get namespace frontend-app -o yaml > frontend-backup.yaml Includes all resources in the namespace
backend-db Database and data processing components kubectl get namespace backend-db -o yaml > backend-backup.yaml Use with etcd backup for full recovery
monitoring Monitoring and logging stack kubectl get namespace monitoring -o yaml > monitoring-backup.yaml Includes Prometheus, Grafana, and Loki

Example: Backup a Specific Namespace

kubectl get namespace frontend-app -o yaml > frontend-backup.yaml

Example: Restore a Specific Namespace

kubectl apply -f frontend-backup.yaml
Note: Ensure the namespace is deleted or cleaned up before restoring to avoid conflicts.
Warning: Backing up namespaces does not include secrets or persistent volumes. Use Velero for full backup capabilities.

7. Excluding Resources

Resource Type Exclusion Method Notes
Deployment metadata.labels.exclude: "true" Label to exclude from backup
StatefulSet metadata.annotations.backup.exclude: "true" Annotation to skip backup
PersistentVolume metadata.annotations.backup.skip: "true" Annotation to skip volume backup
ConfigMap metadata.labels.backup.exclude: "true" Label to exclude from backup
Secret metadata.annotations.backup.exclude: "true" Annotation to skip secret backup

Example: Excluding a Deployment from Backup

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    exclude: "true"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
Note: The exclusion logic depends on the backup tool (e.g., Velero) and its configuration.

8. Using Backup Hooks

Hook Type Description Usage Notes
pre-backup Executed before the backup process starts pre-backup: /bin/bash -c 'echo "Starting backup"' Use for pre-flight checks or cleanup
post-backup Executed after the backup completes post-backup: /bin/bash -c 'echo "Backup completed"' Use for notifications or cleanup
pre-restore Executed before the restore process begins pre-restore: /bin/bash -c 'echo "Preparing for restore"' Use to pause services or set up environment
post-restore Executed after the restore completes post-restore: /bin/bash -c 'echo "Restore completed"' Use to restart services or validate restore

Example: Using Backup Hooks in a CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: backup-cronjob
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup-hook
            image: alpine
            command:
            - /bin/sh
            - -c
            - |
              echo "Starting backup"
              # Backup logic here
              echo "Backup completed"
          restartPolicy: OnFailure
Note: Backup hooks are typically used in conjunction with tools like Velero or custom scripts to automate backup and restore operations.

9. Migrating Across Clusters

Step Description Tools/Commands
1. Identify Source Cluster Ensure the source cluster is stable and accessible. kubectl config current-context
2. Export Resources Export all relevant Kubernetes resources to YAML files. kubectl get all -o yaml > exported-resources.yaml
3. Handle Dependencies Ensure dependencies like ConfigMaps, Secrets, and PersistentVolumes are included. kubectl get configmaps -o yaml, kubectl get secrets -o yaml
4. Validate Exported Resources Verify that all resources are correctly exported and can be applied. kubectl apply --dry-run=client -f exported-resources.yaml
5. Apply to Target Cluster Apply the exported resources to the target cluster. kubectl apply -f exported-resources.yaml

Example: Migrating a Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
Note: Ensure that the target cluster has sufficient resources and compatible versions of Kubernetes to support the migrated workload.
Warning: Migrating across clusters may require adjustments to network policies, service types, or storage configurations.

10. Monitoring Backup Status

Resource Status Field Description Example Value
Backup status.phase Current phase of the backup process Completed
Backup status.conditions List of conditions indicating backup status [{type: "Ready", status: "True"}]
Backup status.backupSize Total size of the backup in bytes 1024000
Restore status.phase Current phase of the restore process Running
Restore status.conditions List of conditions indicating restore status [{type: "Ready", status: "False"}]

Example: Check Backup Status

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: example-backup
status:
  phase: Completed
  conditions:
    - type: Ready
      status: "True"
  backupSize: 1024000

Example: Check Restore Status

apiVersion: velero.io/v1
kind: Restore
metadata:
  name: example-restore
status:
  phase: Running
  conditions:
    - type: Ready
      status: "False"
Note: Use kubectl get backups -n velero or kubectl get restores -n velero to view backup and restore statuses in the cluster.

11. Troubleshooting Backups

Common Backup Issues and Resolution Steps

  1. Backup Fails Due to Resource Limits
    • Check if the backup job runs out of memory or CPU.
    • Increase resource requests/limits in the backup pod configuration.
  2. Backup Job Stuck in Pending State
    • Verify that the required storage class is available and accessible.
    • Ensure sufficient disk space on the storage backend.
  3. Backup Completes but Data Is Incomplete
    • Validate that all required namespaces and resources are included in the backup.
    • Check for any VolumeSnapshot or PersistentVolume issues.
  4. Backup Logs Show Permission Errors
    • Confirm that the service account used by the backup job has appropriate RBAC permissions.
    • Ensure the BackupLocation or storage provider credentials are correctly configured.
Issue Potential Cause Suggested Fix
Backup job fails with timeout Pod running out of memory or CPU Adjust resources.limits in backup job spec
Backup stuck in pending Storage class not found or unavailable Verify StorageClass and its provisioner
Incomplete restore Missing namespace or resource in backup Review backup configuration and include all resources
Permission denied in logs Insufficient RBAC or storage credentials Check service account and secret configurations

Example: Debugging Backup Job Logs

kubectl logs <backup-job-pod-name> -n backup-namespace
Note: Use kubectl describe job <job-name> to get more details about why a job is stuck or failing.

Was this article helpful?

Related Cheat Sheets