Skip to main content
Mapagam

Implementing Multi-Cluster Deployments

1. Understanding Multi-Cluster Patterns

Pattern Description Use Case Notes
Active-Active Deploy the same application across multiple clusters simultaneously, with traffic distributed between them. High availability and load distribution Requires shared storage or state synchronization
Active-Passive One cluster is active, others are standby. Failover occurs if the active cluster becomes unavailable. Disaster recovery and redundancy Typically uses geo-replication or backup strategies
Federated Applications are deployed across clusters with each managing a subset of the data or users. Geographic distribution and scalability Requires consistent state management across clusters
Hybrid Mix of active-active and active-passive deployments across clusters. Flexibility in deployment strategies Can combine different patterns for specific needs

Example: Active-Active Deployment

apiVersion: v1
kind: Service
metadata:
  name: multi-cluster-service
spec:
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 80
  sessionAffinity: ClientIP
Note: Active-active patterns often require external load balancers or service mesh solutions to ensure traffic is routed correctly across clusters.

Key Considerations

  • State synchronization between clusters
  • Network latency and bandwidth constraints
  • Consistent configuration management across clusters
  • Monitoring and alerting for cross-cluster health

2. Managing Kubeconfig Contexts

Context Cluster User Notes
minikube minikube minikube Local development cluster
production production-cluster admin Production environment
staging staging-cluster staging-user Pre-production testing

Example: Switching Contexts

helm install my-app stable/nginx-ingress \
  --namespace production \
  --kube-context production

Example: Using Multiple Contexts

helm repo add stable https://charts.helm.sh/stable
helm install my-app stable/nginx-ingress \
  --namespace staging \
  --kube-context staging
Note: Ensure kubeconfig files are properly configured and accessible for each target cluster.
Warning: Be cautious when deploying to production clusters. Always verify the context and namespace before installation.

3. Using Environment-Specific Values

When deploying Helm charts across multiple environments (e.g., dev, staging, prod), it's crucial to manage configuration values separately for each environment. Helm supports this through values.yaml files and the ability to override values via command-line flags or separate value files.

File Description Usage
values.yaml Default values for the chart Used when no overrides are provided
values-dev.yaml Environment-specific values for development Applied with --values values-dev.yaml
values-prod.yaml Environment-specific values for production Applied with --values values-prod.yaml

Example: Environment-specific values file

replicaCount: 3
image:
  repository: myapp
  tag: v1.2.0
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

Example: Applying environment-specific values

helm install myapp ./my-chart --values values-prod.yaml
Note: Values files can be merged. For example, values-dev.yaml might override only a few fields from the default values.yaml, while keeping others.

Best Practices for Environment-Specific Values

  • Separate configuration files: Keep values-dev.yaml, values-staging.yaml, and values-prod.yaml in version control.
  • Override only what's needed: Use minimal overrides to reduce drift from base configuration.
  • Use --set for one-time overrides: For temporary or ad-hoc changes, use helm install --set key=value.

4. Implementing GitOps Workflow

Example: GitOps with Argo CD and Helm

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-helm-app
spec:
  project: default
  source:
    repoURL: https://github.com/example/my-helm-charts.git
    targetRevision: HEAD
    path: charts/my-app
    helm:
      valueFiles:
        - values-production.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
Field Description Values
apiVersion API version of the Argo CD Application resource argoproj.io/v1alpha1
kind Resource kind Application
metadata.name Name of the Argo CD application String
spec.project Argo CD project to associate with this app default
spec.source.repoURL Git repository URL containing the Helm chart String
spec.source.targetRevision Git revision to sync from (branch, tag, or commit) HEAD
spec.source.path Path to the Helm chart within the repo String
spec.source.helm.valueFiles List of value files to override chart defaults Array of strings
spec.destination.server Kubernetes API server URL https://kubernetes.default.svc
spec.destination.namespace Kubernetes namespace to deploy to String
spec.syncPolicy.automated.prune Whether to prune resources not defined in the chart true, false
spec.syncPolicy.automated.selfHeal Whether to self-heal drift in deployed resources true, false
Note: Argo CD supports various Git providers (GitHub, GitLab, Bitbucket) and can integrate with Helm repositories.

GitOps Workflow Summary

  • Source of truth is a Git repository containing Helm charts and values files
  • Argo CD watches the Git repo for changes
  • Changes are automatically applied to target clusters
  • Sync policy controls how changes are applied (automated or manual)

5. Synchronizing Releases Across Clusters

Feature Description Notes
helm sync helm sync command to synchronize releases across clusters Experimental feature, requires helm diff plugin
release.name Unique identifier for each release across clusters Must be consistent to ensure synchronization
cluster.config Configuration file specifying target clusters Supports multiple cluster definitions in one config

Example: Synchronizing Release Across Clusters

helm sync my-release --source-cluster prod --target-cluster staging
Note: Ensure all clusters have the same Helm chart version to prevent drift.
Warning: Synchronization does not handle configuration differences between clusters.

Key Considerations

  • Use helm diff to preview changes before synchronization
  • Ensure consistent chart versions across clusters
  • Validate cluster-specific configurations before sync

6. Handling Cluster-Specific Configuration

Configuration Type Description Implementation
Values Files YAML files containing configuration values for different environments. 
# values-production.yaml
replicaCount: 3
image:
  tag: "v1.2.3"
Environment Variables Pass configuration via environment variables in deployment manifests. 
env:
- name: DATABASE_URL
  valueFrom:
    secretKeyRef:
      name: db-secret
      key: url
ConfigMaps Store non-sensitive configuration data as key-value pairs. 
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  config.json: |
    {
      "logLevel": "info",
      "maxRetries": 3
    }

Example: Using Values Files for Different Environments

helm install my-app ./my-chart \
  --values values-dev.yaml \
  --set image.tag=latest
Note: Values files can be merged using --values flag multiple times, with later values overriding earlier ones.

Comparison: Values Files vs ConfigMaps

  • Values Files: For chart-wide configuration, managed by Helm.
  • ConfigMaps: For application-level configuration, managed outside Helm.

7. Managing Cross-Cluster Dependencies

Dependency Type Description Handling Method Notes
ConfigMap Shared configuration data across clusters helm dependency add with configMap reference Use helm template to validate before deployment
Secret Secure sensitive data across clusters helm dependency add with secret reference Ensure secret values are encrypted in transit and at rest
Service Inter-cluster communication endpoint kubectl expose or helm install with service config Use externalName or LoadBalancer for cross-cluster access
PersistentVolume Shared storage between clusters helm install with PersistentVolumeClaim Requires compatible storage systems across clusters

Example: Using ConfigMap Across Clusters

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  config.yaml: |
    database:
      host: "db-cluster-1"
      port: 5432

Example: Referencing Secret in Chart

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque
data:
  password: "ZGVmYXVsdF9wYXNz"
Note: When deploying to multiple clusters, ensure that the same ConfigMap or Secret is available in each cluster's namespace.
Warning: Do not hardcode sensitive values in chart templates. Use helm install --set or --values to inject values at deployment time.

Best Practices for Cross-Cluster Dependencies

  • Use Helm dependencies to manage shared resources
  • Validate configurations before deploying across clusters
  • Encrypt secrets in transit and at rest
  • Ensure consistent namespace usage across clusters

8. Implementing Blue-Green Deployments

Feature Description Notes
blue-green deployment Deploying new version to a separate cluster or namespace, then switching traffic Ensures zero-downtime and rollback capability
traffic routing Using service mesh or ingress controller to route traffic between versions Can be done via labels, annotations, or external tools like Istio
version switch Switching traffic from old to new version using service selectors or routing rules Can be automated with CI/CD pipelines or manual intervention

Example: Blue-Green Deployment with Helm

apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    app: app-blue
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app-blue
  template:
    metadata:
      labels:
        app: app-blue
    spec:
      containers:
        - name: app
          image: myapp:v1.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-green
spec:
  replicas: 3
  selector:
    matchLabels:
      app: app-green
  template:
    metadata:
      labels:
        app: app-green
    spec:
      containers:
        - name: app
          image: myapp:v2.0

Workflow: Blue-Green Deployment Steps

  1. Deploy new version (v2.0) to "green" environment
  2. Run tests on green version
  3. Switch traffic from "blue" to "green" using service selector
  4. Optionally scale down or delete old version (blue)

Comparison: Blue-Green vs Rolling Updates

  • Blue-Green: Two separate environments, zero downtime
  • Rolling Update: Gradual replacement of pods, less downtime

9. Managing Canary Releases

Field Description Example
canary.enabled Enables canary deployment strategy true
canary.weight Percentage of traffic to send to canary 10
canary.replicas Number of replicas for canary deployment 1
canary.strategy Deployment strategy (e.g., RollingUpdate) RollingUpdate

Example: Canary Release Configuration

canary:
  enabled: true
  weight: 10
  replicas: 1
  strategy:
    type: RollingUpdate
Note: Canary releases help reduce risk by gradually shifting traffic to a new version.
Warning: Ensure proper monitoring and rollback mechanisms are in place for canary deployments.

10. Monitoring Multi-Cluster Deployments

Example: Prometheus Configuration for Multi-Cluster

scrape_configs:
  - job_name: 'helm-deployments'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: $1:9090
Configuration Description Values Notes
job_name Unique identifier for the job 'helm-deployments' Used to group metrics from this job
kubernetes_sd_configs Defines how to discover targets in Kubernetes [{role: pod}] Scrapes pods that match the criteria
relabel_configs Rules to modify labels before scraping [{source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape], action: keep, regex: true}] Filters pods based on annotations

Key Monitoring Concepts

  • Service Discovery in Prometheus allows automatic detection of pods or services to monitor.
  • Annotations are used to configure scraping behavior per pod.
  • Relabeling allows modification of labels before metrics are scraped.

Monitoring Approaches

  • Centralized Monitoring: All clusters report to one Prometheus instance
  • Distributed Monitoring: Each cluster has its own Prometheus instance
  • Hybrid Approach: Centralized for alerts, distributed for metrics
Note: Ensure that the Prometheus instance has proper RBAC permissions to access pods in all clusters.
Warning: Multi-cluster monitoring can increase latency and resource usage. Monitor the performance of your Prometheus instances.

Was this article helpful?

Related Cheat Sheets