Skip to main content
Mapagam

Working with Custom Resource Definitions

1. Understanding CRD Handling

Example: Basic CRD Definition

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule:
                  type: string
                  pattern: '^[0-9/ *,-]+
Property Description Values
apiVersion API version of the CRD apiextensions.k8s.io/v1
kind Kind of the CRD object CustomResourceDefinition
metadata.name Unique name for the CRD crontabs.stable.example.com
spec.group API group for the CRD stable.example.com
spec.versions.name Name of the version v1
spec.scope Scope of the CRD (Namespaced or Cluster) Namespaced, Cluster
spec.names.plural Plural form of the resource name crontabs
spec.names.singular Singular form of the resource name crontab
spec.names.kind Kind of the CRD CronTab
Note: Helm charts can reference CRDs via crd templates in the templates/ directory.

CRD Handling in Helm Charts

  • Helm charts can define CRDs using crd templates in the templates/ directory.
  • CRDs are installed before any chart resources that depend on them.
  • CRDs can be defined in crds/ directory and referenced via crd templates.
  • CRDs are not managed by Helm's release lifecycle, but can be updated via chart upgrades.

2. Installing CRDs with Charts

Field Description Example
crds Directory containing CRD definitions crds/
install Whether to install CRDs during chart installation true

Example: CRD Installation in Chart

apiVersion: v2
name: my-app
version: 1.0.0
...
crds:
  - crds/my-crd.yaml
Note: CRDs must be installed before resources using them.
Warning: Helm does not manage CRD updates after initial install.

3. Creating CRD Templates

Property Value Description
apiVersion apiextensions.k8s.io/v1 API version of the CRD resource
kind CustomResourceDefinition Kind of the CRD resource
metadata.name my-crd.example.com Name of the CRD, must follow DNS subdomain format
spec.versions.name v1 Name of the version, used in API calls
spec.versions.schema.openAPIV3Schema.properties.spec object Schema definition for the spec field of the custom resource

Example: Basic CRD Template

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: my-crd.example.com
spec:
  group: example.com
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                  minimum: 0
Note: The openAPIV3Schema is required for validation and must be a valid OpenAPI v3 schema.
Warning: CRDs must be installed before any custom resources using them can be created.

4. Managing CRD Versions

Version Schema Changes Notes
v1 spec.versions array with served and storage flags Latest stable version with full backward compatibility support
v1beta1 Legacy version with spec.version field DEPRECATED in Kubernetes 1.16+, removed in 1.22+

Example: CRD v1 Schema Definition

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                schedule:
                  type: string
      served: true
      storage: true
Note: When migrating from v1beta1 to v1, ensure all spec.version fields are updated to spec.versions array format.
Warning: CRDs using v1beta1 will not be supported in Kubernetes 1.22+. Migrate to v1 schema before upgrading.

5. Handling CRD Upgrades

Upgrade Type Description Notes
Major Version Upgrade Changing from v1 to v2, or any non-backward-compatible change Requires manual intervention and careful migration planning
Minor Version Upgrade Adding new fields or updating existing ones with backward compatibility Can be handled automatically by Helm if using proper schema validation
Patch Version Upgrade Fixes or minor updates without changing structure Generally safe to upgrade without additional steps

Example: Upgrading CRD with New Field

# Before upgrade
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrds.example.com
spec:
  group: example.com
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            spec:
              type: object
              properties:
                name:
                  type: string

# After upgrade
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrds.example.com
spec:
  group: example.com
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          properties:
            spec:
              type: object
              properties:
                name:
                  type: string
                description:
                  type: string
Note: Helm does not automatically update CRDs in the cluster. You must manually apply updated CRD definitions or use helm upgrade --set crds=true to ensure CRDs are updated.

CRD Upgrade Workflow

  1. Review the changes in the new CRD version
  2. Update your Helm chart's CRD template to match the new schema
  3. Run helm upgrade --set crds=true to apply the updated CRD
  4. Verify that existing CRs still validate against the new schema
  5. Update any dependent resources or controllers to handle new fields if needed

CRD Upgrade Approaches

  • Manual Update: Apply CRD YAML directly using kubectl apply
  • Helm Managed: Include CRD in chart and use --set crds=true
  • Operator-Based: Use an operator to manage CRD lifecycle and upgrades

6. Avoiding CRD Deletion on Uninstall

Example: Preventing CRD Deletion

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: mycrds.example.com
  annotations:
    helm.sh/resource-policy: keep
spec:
  group: example.com
  versions:
    - name: v1
      schema:
        openAPIV3Schema:
          type: object
Annotation Value Description
helm.sh/resource-policy keep Prevents Helm from deleting the CRD during uninstall
Note: The helm.sh/resource-policy annotation must be set on the CRD's metadata to prevent deletion during helm uninstall.
Warning: If a CRD is deleted while resources exist, those resources become orphaned and inaccessible.

7. Validating CRD Schemas

Example: Basic CRD Schema Validation

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: prometheuses.monitoring.coreos.com
spec:
  group: monitoring.coreos.com
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            required:
            - resources
            properties:
              resources:
                type: array
                items:
                  type: object
                  required:
                  - name
                  properties:
                    name:
                      type: string
                      pattern: "^[a-zA-Z][a-zA-Z0-9-]*$"
                    port:
                      type: integer
                      minimum: 1
                      maximum: 65535
                additionalProperties: false
            additionalProperties: false
Field Description Validation Rules
spec.versions.schema.openAPIV3Schema OpenAPI v3 schema for validating CR instances Must be a valid OpenAPI v3 schema object
properties.name.pattern Regex pattern for resource name validation Must match ^[a-zA-Z][a-zA-Z0-9-]*$
properties.port.minimum Minimum allowed port value Must be >= 1
properties.port.maximum Maximum allowed port value Must be <= 65535
Note: The openAPIV3Schema field uses the OpenAPI v3 schema format to define validation rules for Custom Resources. It supports all standard OpenAPI validation keywords like pattern, minimum, and maximum.
Warning: Invalid schema definitions will cause CRD creation to fail. Always validate schemas before applying them to the cluster.

8. Using CRDs with Operators

Custom Resource Definitions (CRDs) are often used in conjunction with Kubernetes Operators to extend the Kubernetes API and automate operations for custom resources.

Component Description Usage
CRD Defines a new custom resource type in the Kubernetes API Used to create custom resources that extend Kubernetes functionality
Operator Controller that manages custom resources defined by CRDs Automates deployment, scaling, and management of custom resources

Example: Operator Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-operator
  template:
    metadata:
      labels:
        app: my-operator
    spec:
      containers:
      - name: operator
        image: my-operator-image

Example: CRD Definition

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: myresources.example.com
spec:
  group: example.com
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              replicas:
                type: integer
Note: Operators use CRDs to define custom resources and implement controllers that watch for changes to these resources.

Key Concepts

  • CRDs define custom resource types that extend Kubernetes API
  • Operators manage custom resources using controllers
  • CRDs and Operators work together to automate complex operations

9. Testing CRD Installation

Example: Verify CRD Installation

kubectl get crd my-crd.example.com
Command Description Output
kubectl get crd Lists all installed CRDs Displays CRD names and versions
kubectl describe crd <name> Shows detailed CRD information Includes spec, status, and validation rules
Note: Ensure the CRD is fully registered before deploying resources that use it.

Example: Validate CRD with Sample Resource

apiVersion: example.com/v1
kind: MyCustomResource
metadata:
  name: test-resource
spec:
  replicas: 2

Testing Workflow

  1. Create a test CRD using kubectl apply
  2. Verify CRD status with kubectl get crd
  3. Apply a sample custom resource using the CRD
  4. Check that the resource is created successfully

10. Troubleshooting CRD Issues

Example: Invalid CRD Schema

{
  "apiVersion": "apiextensions.k8s.io/v1",
  "kind": "CustomResourceDefinition",
  "metadata": {
    "name": "myresources.example.com"
  },
  "spec": {
    "group": "example.com",
    "versions": [
      {
        "name": "v1",
        "served": true,
        "storage": true,
        "schema": {
          "openAPIV3Schema": {
            "type": "object",
            "properties": {
              "spec": {
                "type": "object",
                "properties": {
                  "invalidField": {
                    "type": "string"
                  }
                },
                "required": ["invalidField"]
              }
            }
          }
        }
      }
    ]
  }
}
Issue Description Solution
Invalid Schema CRD schema does not conform to OpenAPI v3 spec Validate using helm template or kubectl apply --dry-run=client
Missing Required Fields CRD schema defines required fields that aren't present in instances Add missing fields to CR instances or update CRD schema
Version Mismatch CRD version used in chart doesn't match installed CRD Ensure CRD versions align between chart and cluster
Note: Use kubectl get crd <crd-name> -o yaml to inspect CRD schema and status.
Warning: Modifying CRDs after resources exist can lead to data loss or unexpected behavior.

Common CRD Troubleshooting Steps

  • Check CRD status with kubectl describe crd <name>
  • Validate CRD schema using kubectl apply --dry-run=client
  • Ensure CR instances match the defined schema
  • Verify Helm chart and installed CRD versions are compatible
status: type: object properties: lastTime: type: string format: date-time scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab

Was this article helpful?

Related Cheat Sheets