Skip to main content
Mapagam

Managing Multi-Tenancy

1. Creating Tenant Namespaces

Property Description Example
metadata.name Name of the namespace tenant-a
metadata.labels Labels to identify the tenant {"tenant.id": "a", "environment": "production"}
spec Optional specification for namespace features {"finalizers": ["kubernetes"]}

Example: Create a Tenant Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    tenant.id: a
    environment: production
Note: Namespaces are scoped to a single Kubernetes cluster and do not provide resource isolation across clusters.

2. Setting Resource Quotas

Resource Scope Description Example
limits.cpu cpu Maximum CPU usage per pod 2
limits.memory memory Maximum memory usage per pod 512Mi
requests.cpu cpu Minimum CPU requested per pod 0.5
requests.memory memory Minimum memory requested per pod 256Mi

Example: Resource Quota Definition

apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
spec:
  hard:
    cpu: "2"
    memory: 512Mi
    pods: "10"
Note: Resource quotas are enforced at the namespace level and apply to all pods within that namespace.
Warning: Setting quotas too low can cause pods to be evicted or fail to start.

3. Configuring Limit Ranges

Example: Define LimitRange for Namespace

apiVersion: v1
kind: LimitRange
metadata:
  name: mem-limit-range
spec:
  limits:
  - default:
      memory: 512Mi
    defaultRequest:
      memory: 256Mi
    type: Container
Field Description Values
spec.limits[].type Resource type to apply limits to Container, PersistentVolumeClaim
spec.limits[].default Default resource limit for each container cpu, memory
spec.limits[].defaultRequest Default resource request for each container cpu, memory
Note: LimitRanges are enforced at pod creation time and cannot be changed after a pod is created.
Warning: LimitRanges do not apply to Pod objects directly, but only to containers within pods in the namespace where the LimitRange is defined.

4. Implementing RBAC Isolation

Resource Permission Description
Role get, list, watch Read access to resources within a namespace
ClusterRole get, list, watch Read access to cluster-wide resources
RoleBinding bind Assigns roles to users or groups within a namespace
ClusterRoleBinding bind Assigns cluster roles to users or groups across the entire cluster

Example: Creating a Role for Pod Access

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: production
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]

Example: Binding Role to User

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: User
  name: alice
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
Note: RBAC rules are additive; permissions granted to a user or group are cumulative across all bindings.

5. Using Network Policies

Example: Basic Network Policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
Field Description Values
apiVersion API version of the NetworkPolicy networking.k8s.io/v1
kind Kind of the resource NetworkPolicy
metadata.name Name of the NetworkPolicy String
spec.podSelector Selector to identify pods to which this policy applies LabelSelector
spec.policyTypes Types of policy (Ingress, Egress) ["Ingress", "Egress"]
spec.ingress Rules for incoming traffic List<NetworkPolicyIngressRule>
spec.egress Rules for outgoing traffic List<NetworkPolicyEgressRule>
Note: Network policies are enforced by the CNI plugin. Ensure your cluster supports network policies (e.g., Calico, Cilium).

Example: Deny All Ingress

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}
  policyTypes:
    - Ingress

Example: Allow From Specific Namespace

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-frontend-namespace
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: frontend-namespace
      ports:
        - protocol: TCP
          port: 8080

6. Configuring Pod Security

Policy Description Configuration Notes
Pod Security Standards Enforce security policies for pods in a namespace pod-security.kubernetes.io/enforce=privileged Values: enforce, audit, warn
Enforce Level Level of enforcement for pod security enforce=restricted Options: privileged, baseline, restricted
Namespace Label Label to apply pod security policy to a namespace pod-security.kubernetes.io/enforce=baseline Applied to namespace metadata

Example: Applying Pod Security Standard

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: baseline

Example: Enforcing Restricted Policy

apiVersion: v1
kind: Namespace
metadata:
  name: dev
  labels:
    pod-security.kubernetes.io/enforce: restricted
Note: Pod Security Admission is available in Kubernetes v1.23+ and requires the PodSecurity admission controller to be enabled.
Warning: Setting enforce=privileged allows any pod to run with elevated privileges and should be used carefully.
Enforcement Level Description Restrictions Use Case
privileged Allows any pod to run with full privileges No restrictions System-level namespaces or legacy apps
baseline Allows minimal pod configuration Disallows hostPath, hostNetwork, and hostPID Default for most namespaces
restricted Most restrictive policy Disallows privileged containers, host namespaces, and unsafe capabilities High-security environments

7. Using Virtual Clusters

Feature Description Usage
Virtual Cluster A virtual cluster is a logical grouping of resources within a Kubernetes cluster that isolates workloads and configurations. Used to provide multi-tenancy and resource isolation in a shared cluster environment.
Namespace Isolation Each virtual cluster can be associated with one or more namespaces to isolate workloads. Enforced via RBAC and namespace-level resource quotas.
Resource Quotas Limit the amount of resources (CPU, memory) that can be consumed by a virtual cluster. Configured via ResourceQuota objects in each namespace.

Example: Creating a Virtual Cluster with Namespaces

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-a
  labels:
    virtual-cluster: "tenant-a"
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota-tenant-a
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
Note: Virtual clusters are not a built-in Kubernetes feature but can be implemented using namespaces, RBAC, and resource quotas.

Virtual Clusters vs. Dedicated Clusters

  • Virtual Clusters: Shared cluster with isolation via namespaces and quotas.
  • Dedicated Clusters: Isolated clusters for each tenant, offering stronger isolation.

8. Implementing Hierarchical Namespaces

Property Description Example
namespace.name Name of the namespace production
namespace.labels Labels to organize namespaces {"environment": "prod", "team": "backend"}
namespace.annotations Annotations for metadata {"description": "Production backend services"}

Example: Creating a Hierarchical Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    environment: prod
    team: backend
  annotations:
    description: "Production backend services"
Note: Hierarchical namespaces can be managed using label and annotation keys to group related namespaces under a common structure.

Best Practices for Hierarchical Namespaces

  • Use consistent naming conventions across teams.
  • Apply labels to enable filtering and grouping of namespaces.
  • Implement RBAC policies to control access at the namespace level.

9. Monitoring Tenant Resources

Resource Metric Description Units
CPU Usage container_cpu_usage_seconds_total Total CPU time consumed by containers seconds
Memory Usage container_memory_usage_bytes Current memory usage in bytes bytes
Network I/O container_network_receive_bytes_total Total bytes received over network interface bytes
Disk I/O container_fs_reads_bytes_total Total bytes read from disk bytes

Example: Querying CPU Usage for a Namespace

query="container_cpu_usage_seconds_total{namespace=\"tenant-a\"}"
prometheus-cli query "$query"
Note: Ensure that namespace labels are consistently applied to all resources in a tenant for accurate monitoring.

Key Metrics to Monitor

  • CPU Usage - Track for overconsumption or throttling.
  • Memory Usage - Watch for out-of-memory (OOM) events.
  • Network I/O - Identify potential bandwidth bottlenecks.
  • Disk I/O - Monitor for storage contention or slow disk access.

10. Implementing Cost Allocation

Resource Cost Allocation Method Description
Namespace label Assign cost tags to namespaces using labels for billing purposes
ResourceQuota limit Set resource limits per namespace to control consumption and cost
HorizontalPodAutoscaler autoscale Automatically scale pods based on CPU or memory usage to optimize costs

Example: Labeling Namespaces for Cost Tracking

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    cost-center: engineering
    owner: team-alpha

Example: Setting Resource Limits

apiVersion: v1
kind: ResourceQuota
metadata:
  name: quota
  namespace: production
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi

Example: Configuring Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
Note: Cost allocation in Kubernetes requires integration with cloud provider billing systems or third-party tools like kube-cost for detailed cost tracking.
Warning: Misconfigured resource limits can lead to performance degradation or application downtime.

Was this article helpful?

Related Cheat Sheets