Managing External Secrets

1. Installing External Secrets Operator

Example: Install External Secrets Operator using Helm

helm repo add external-secrets https://external-secrets.github.io/external-secrets
helm install external-secrets external-secrets/external-secrets --namespace external-secrets --create-namespace

Component	Description	Version
Operator	Manages external secrets in Kubernetes	1.12.0
CRDs	Custom Resource Definitions for SecretStore and ExternalSecret	v1beta1

Note: Ensure the namespace external-secrets exists or is created during installation.

Warning: The operator requires RBAC permissions to manage secrets in the target namespace.

Installation Steps

Add the Helm repository for External Secrets
Install the operator using Helm chart
Verify installation in the external-secrets namespace

2. Creating SecretStore

Field	Description	Example Value
apiVersion	API version of the SecretStore resource	`external-secrets.io/v1beta1`
kind	Kind of the resource (must be SecretStore)	`SecretStore`
metadata.name	Name of the SecretStore resource	`my-secret-store`
spec.provider	Provider configuration for external secret management	`aws, azure, gcp, vault`

Example: AWS SecretStore

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: access-key-id
          secretAccessKeySecretRef:
            name: aws-credentials
            key: secret-access-key

Example: Vault SecretStore

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-secret-store
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret/data/myapp
      auth:
        tokenSecretRef:
          name: vault-token
          key: token

3. Creating ClusterSecretStore

Field	Description	Example Value
apiVersion	API version of the resource	`external-secrets.io/v1beta1`
kind	Kind of the resource	`ClusterSecretStore`
metadata.name	Name of the ClusterSecretStore	`my-cluster-secret-store`
spec.provider	Secret provider configuration	`{ "aws": { ... } }`

Example: Creating a ClusterSecretStore for AWS Secrets Manager

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-credentials
            key: access-key-id
          secretAccessKeySecretRef:
            name: aws-credentials
            key: secret-access-key

Note: The external-secrets.io CRD must be installed in the cluster before creating ClusterSecretStore resources.

4. Creating ExternalSecret Resource

Field	Description	Example Value
apiVersion	API version of the ExternalSecret resource	`external-secrets.io/v1beta1`
kind	Kind of the Kubernetes resource	`ExternalSecret`
metadata.name	Name of the ExternalSecret resource	`my-secret`
spec.secretStoreRef.name	Name of the SecretStore to use for fetching secrets	`my-secret-store`
spec.data	Mapping of secret keys to their external sources	`[{ secretKey: "password", remoteRef: { key: "prod/db/password" } }]`

Example: Creating an ExternalSecret

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-secret
spec:
  secretStoreRef:
    name: my-secret-store
  data:
    - secretKey: password
      remoteRef:
        key: prod/db/password

5. Syncing Secrets Automatically

Field	Description	Example
secretStoreRef.name	Name of the SecretStore to use for fetching secrets	`my-secret-store`
secretStoreRef.kind	Kind of the SecretStore (e.g., `ClusterSecretStore`)	`ClusterSecretStore`
refreshInterval	How often to refresh the secret (e.g., `1h`)	`5m`
retryInterval	Time to wait before retrying failed sync (e.g., `10s`)	`30s`

Example: ExternalSecret Configuration

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example-secret
spec:
  refreshInterval: "1h"
  secretStoreRef:
    name: my-secret-store
    kind: ClusterSecretStore
  target:
    name: internal-secret
  data:
  - secretKey: password
    remoteRef:
      key: db/password

Note: The refreshInterval determines how often the secret is synced from the external store.

Warning: Ensure that secretStoreRef points to a valid and accessible SecretStore.

6. Using Secret Templates

Property	Description	Example
template.name	Name of the secret template	`db-credentials`
template.type	Type of secret (e.g., `Opaque`, `tls`)	`Opaque`
template.data	Map of key-value pairs for secret data	`{ "username": "admin", "password": "secret123" }`

Example: Define a Secret Template

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  username: YWRtaW4=
  password: c2VjcmV0MTIz

Note: The data field should contain base64-encoded values.

Example: Using Template in Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        envFrom:
        - secretRef:
            name: db-credentials

7. Configuring Secret Keys

Property	Description	Values	Notes
key	Specifies the key name in the secret	`string`	Required field for each key-value pair
name	Alias for the key in the secret	`string`	Optional, defaults to key name
valueFrom	Source of the secret value	`object`	Can reference external secrets or config maps

Example: Secret Key Configuration

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  database-password: "cm9vdHBhc3N3b3Jk"
  api-key: "MTIzNDU2Nzg5MGFiY2RlZg=="

Note: The values in the data field must be base64-encoded.

Example: Using Secret Keys in Pod

apiVersion: v1
kind: Pod
metadata:
  name: secret-pod
spec:
  containers:
  - name: app-container
    image: nginx
    env:
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: database-password
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secret
  volumes:
  - name: secret-volume
    secret:
      secretName: my-secret

8. Rotating Secrets

Secret	Rotation Method	Notes
kubernetes.io/tls	Manual regeneration	Requires updating the secret with new certificate and key
Opaque	Manual update or automated tool	Can be updated via CLI or API with new values
Opaque (with external manager)	Automated rotation	Uses tools like Vault, Sealed Secrets, or external secret managers

Example: Manual Rotation of Opaque Secret

kubectl delete secret my-secret
kubectl create secret generic my-secret \
  --from-literal=token=NEW_TOKEN_VALUE \
  --from-literal=password=NEW_PASSWORD

Example: Using Sealed Secrets for Rotation

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: my-sealed-secret
spec:
  encryptedData:
    token: <encrypted_value>
    password: <encrypted_value>

Note: When rotating secrets, ensure that all pods using the secret are restarted to pick up the new values.

9. Configuring Data Transformation

Property	Description	Values	Notes
dataTransform	Defines transformation rules for secret data	`string`	Used with external secret providers like AWS Secrets Manager or Vault
transformations	List of transformation operations to apply	`array`	Each item is an object with `type` and `params`

Example: Transforming Secret Data

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example-secret
spec:
  data:
    - secretKey: password
      remoteRef:
        key: /myapp/password
        transformation:
          - type: jsonPath
            params:
              path: $.password
          - type: base64decode

Note: Transformation rules are applied in order. Each transformation can modify the data before it reaches the Kubernetes secret.

Warning: Ensure that transformations do not inadvertently expose sensitive data or cause decoding errors.

10. Troubleshooting Secret Sync

Issue	Possible Cause	Solution
Secrets not syncing to pods	Incorrect secret name or namespace	Verify secret exists in correct namespace and is referenced properly in pod spec
Secrets not updating in pods	Cache or stale data in pod	Restart pod or use rolling update to refresh secret mount
Permission denied accessing secret	RBAC misconfiguration or missing role bindings	Ensure service account has appropriate permissions to get secrets in namespace
Secrets missing from pod filesystem	Mount path incorrect or secret not mounted	Check volume mount configuration in pod spec and ensure secret is properly referenced
Secrets appear as empty files	Incorrect secret data format or key name mismatch	Validate secret keys and ensure they match what's expected by application

Example: Check Secret Status

kubectl get secrets -n <namespace>
kubectl describe secret <secret-name> -n <namespace>

Example: Describe Pod Volume Mount

kubectl describe pod <pod-name> -n <namespace>

Note: Use kubectl logs to check for any errors in secret injection or mounting components like external-secrets-operator.

Warning: Never store sensitive data in pod specs directly; always use external secret managers.