Working with Multi-Cluster Federation

1. Understanding Multi-Cluster Patterns

Pattern	Description	Use Case	Tools
Active-Active	Multiple clusters actively serving traffic with load balancing	High availability, geographic distribution	Kubernetes Federation, Istio, Traefik
Active-Passive	One cluster is primary, others are standby for failover	Disaster recovery, redundancy	Kubernetes Federation, Velero
Hybrid	Mix of on-prem and cloud clusters with shared control plane	Hybrid cloud strategy, compliance	Kubernetes Federation, Rancher
Multi-Tenant	Isolated clusters per tenant or team	Shared infrastructure, security isolation	Kubernetes Namespaces, RBAC

Example: Active-Active Cluster Setup

# Example config for multi-cluster active-active setup
apiVersion: v1
kind: Service
metadata:
  name: multi-cluster-service
spec:
  selector:
    app: frontend
  ports:
    - port: 80
      targetPort: 80
  sessionAffinity: ClientIP

Note: Active-Active patterns require consistent state management and cross-cluster communication.

Key Considerations

Network connectivity between clusters
Data synchronization and consistency
Resource quotas and limits across clusters
Security policies and identity management

2. Installing KubeFed

Example: Install KubeFed using kubectl

helm repo add kubefed https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
helm install kubefed kubefed/kubefed -n kubefed-system --create-namespace

Component	Description	Version
KubeFed Controller	Manages federation of resources across clusters	`v0.12.0`
KubeFed CRDs	Custom Resource Definitions for federation	`v0.12.0`

Note: Ensure that the kubefed-system namespace exists before installing KubeFed.

Warning: The installation requires cluster-admin privileges.

Installation Steps

Install Helm if not already installed
Add the KubeFed Helm repository
Install KubeFed using Helm with appropriate namespace

3. Creating Federated Clusters

Example: Federated Cluster Configuration

apiVersion: federation.k8s.io/v1
kind: FederatedCluster
metadata:
  name: production-cluster
spec:
  clusterRef:
    name: production-cluster
  apiEndpoint:
    host: "https://production.example.com"
    port: 443
  secretRef:
    name: cluster-secret

Field	Description	Required
apiVersion	API version of the FederatedCluster resource	Yes
kind	Kind of the resource (FederatedCluster)	Yes
metadata.name	Name of the federated cluster	Yes
spec.clusterRef.name	Name of the actual cluster in the federation control plane	Yes
spec.apiEndpoint.host	Host address of the cluster API server	Yes
spec.apiEndpoint.port	Port number of the cluster API server (default 443)	No
spec.secretRef.name	Name of the secret containing cluster credentials	Yes

Note: The federation.k8s.io/v1 API version is used for Kubernetes 1.16+.

Warning: Ensure the cluster secret has proper RBAC permissions to access the target cluster.

4. Deploying Federated Resources

Resource Type	Description	Deployment Method	Notes
FederatedDeployment	Manages Deployment objects across multiple clusters	`kubectl apply -f deployment.yaml`	Requires federation control plane
FederatedService	Manages Service objects across clusters	`kubectl apply -f service.yaml`	Supports cross-cluster load balancing
FederatedConfigMap	Manages ConfigMap objects across clusters	`kubectl apply -f configmap.yaml`	Used for centralized configuration
FederatedSecret	Manages Secret objects across clusters	`kubectl apply -f secret.yaml`	Securely distributes secrets

Example: Deploying a Federated Deployment

apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: my-app-deployment
spec:
  template:
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: my-app
      template:
        metadata:
          labels:
            app: my-app
        spec:
          containers:
          - name: my-app-container
            image: nginx:latest
            ports:
            - containerPort: 80

Note: Ensure that the federation control plane is running and configured before deploying federated resources.

Warning: Federated resources must be defined in a namespace that is enabled for federation.

5. Configuring Placement Policies

Policy Type	Description	Configuration Example	Notes
clusterAffinity	Defines preferred clusters based on labels	`{ "preferredDuringSchedulingIgnoredDuringExecution": [ { "weight": 100, "preference": { "matchLabels": { "environment": "production" } } } ] }`	Used in `Placement` rules to schedule resources on clusters matching labels
clusterSelector	Matches clusters using label selectors	`{ "clusterSelector": { "matchLabels": { "region": "us-west" } } }`	Allows targeting clusters by region, zone, or custom labels
spreadConstraints	Ensures resources are spread across clusters	`{ "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone" }`	Prevents over-concentration of resources in one cluster

Example: Placement Policy with Cluster Affinity

apiVersion: multicluster.x-k8s.io/v1alpha1
kind: Placement
metadata:
  name: my-placement
spec:
  clusterAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchLabels:
            environment: production

Note: Placement policies are part of the cluster-api and cluster-api-provider ecosystem, not core Kubernetes.

6. Implementing Cross-Cluster Discovery

Feature	Description	Implementation	Notes
Federation API	Enables cross-cluster service discovery and load balancing	`apiVersion: federation/v1beta1 kind: FederatedService metadata: name: my-service spec: template: spec: ports: - port: 80 targetPort: 8080`	Requires federation control plane setup
Service Discovery	Uses DNS or service registry for cross-cluster resolution	`apiVersion: v1 kind: Service metadata: name: cross-cluster-service spec: selector: app: frontend ports: - port: 80 targetPort: 8080`	Supports DNS-based discovery in multi-cluster environments

Example: Federated Service Configuration

apiVersion: federation/v1beta1
kind: FederatedService
metadata:
  name: frontend-service
spec:
  template:
    spec:
      ports:
      - port: 80
        targetPort: 8080
  placement:
    clusters:
    - name: cluster-1
    - name: cluster-2

Note: Cross-cluster discovery requires proper federation setup and cluster registration.

Warning: Ensure all clusters are in the same federation domain for proper discovery.

7. Using Multi-Cluster Ingress

Field	Description	Example
ingressClassName	Specifies the IngressClass to be used for this Ingress resource	`nginx`
rules.host	Hostname for the rule	`api.example.com`
rules.http.paths.path	Path pattern to match requests	`/api/v1/`
rules.http.paths.backend.service.name	Name of the Kubernetes Service to forward traffic to	`backend-service`
rules.http.paths.backend.service.port.number	Port number of the backend service	`8080`

Example: Basic Multi-Cluster Ingress Configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi-cluster-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /api/v1/
        pathType: Prefix
        backend:
          service:
            name: backend-service
            port:
              number: 8080

Note: Ensure that the Ingress controller is properly configured across all clusters to support multi-cluster routing.

Warning: Misconfiguring Ingress rules can lead to service unavailability or incorrect traffic routing.

Key Considerations for Multi-Cluster Ingress

Ensure consistent IngressClass across clusters
Validate backend service names and ports are identical in each cluster
Use pathType Prefix for better compatibility with multi-cluster setups
Monitor cross-cluster network policies to avoid connectivity issues

8. Configuring Context Switching

Context Field	Description	Example Value
name	Name of the context	`production-cluster`
cluster	Name of the cluster to use	`prod-cluster`
user	Name of the user to authenticate as	`admin-user`

Example: Define a new context in kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <cert-data>
    server: https://prod-cluster.example.com
  name: prod-cluster
contexts:
- context:
    cluster: prod-cluster
    user: admin-user
  name: production-cluster
current-context: production-cluster
users:
- name: admin-user
  user:
    client-certificate-data: <cert-data>
    client-key-data: <key-data>

Note: The current-context field determines which context is active by default when running kubectl commands.

Example: Switch context using kubectl

kubectl config use-context production-cluster

Example: View current context

kubectl config current-context

9. Implementing Disaster Recovery

Component	Description	Recovery Strategy
etcd	Primary storage for cluster state	Regular backups with `etcdctl backup`; restore from snapshot on failure
Persistent Volumes	Storage for stateful applications	Use snapshots or replication to external storage (e.g., AWS EBS, GCP PD)
ConfigMaps & Secrets	Configuration and sensitive data	Store in version-controlled Git; restore via `kubectl apply`
Ingress Controllers	External access to services	Recreate with `kubectl apply` from manifest files

Example: Backup etcd

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cert=/etc/ssl/etcd/ssl/node-1.pem \
  --key=/etc/ssl/etcd/ssl/node-1-key.pem \
  --cacert=/etc/ssl/etcd/ssl/ca.pem \
  snapshot save /backup/snapshot.db

Example: Restore etcd from Snapshot

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cert=/etc/ssl/etcd/ssl/node-1.pem \
  --key=/etc/ssl/etcd/ssl/node-1-key.pem \
  --cacert=/etc/ssl/etcd/ssl/ca.pem \
  snapshot restore /backup/snapshot.db

Disaster Recovery Steps

Identify the root cause of failure (e.g., etcd outage, node crash)
Restore cluster state from backups
Recreate all resources using manifests or GitOps tools
Validate application availability and data integrity

Warning: Always test disaster recovery procedures in a non-production environment before relying on them in production.

10. Monitoring Federated Clusters

Component	Description	Monitoring Approach
Federated Metrics	Aggregated metrics from multiple clusters	Use Prometheus with federation or Thanos for cross-cluster metric collection
Cluster Status	Overall health and status of each cluster in the federation	Implement custom controllers or use tools like Kiali for visual cluster status
Resource Utilization	Memory, CPU, and storage usage across clusters	Deploy metrics-server or Prometheus Node Exporter on each cluster
Service Mesh Observability	Tracing and monitoring of services across clusters	Use Istio with Jaeger or Grafana for service mesh observability

Example: Setting up Prometheus Federation for Multi-Cluster Metrics

# prometheus-federate.yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "federate"
    scrape_interval: 15s
    honor_labels: true
    metrics_path: "/federate"
    params:
      match[]:
        - "{__name__=~".+"}"
    static_configs:
      - targets:
          - "prometheus-cluster1:9090"
          - "prometheus-cluster2:9090"

Note: Ensure that all clusters have compatible Prometheus versions and that the federation endpoint is accessible from the central Prometheus instance.

Monitoring Tools Comparison

Prometheus + Thanos: Scalable, federated metrics with long-term storage
Grafana + Loki: Log aggregation and query with rich dashboards
Kiali: Service mesh observability and Istio integration
Elastic Stack (ELK): Centralized logging and analytics