Skip to main content
Mapagam

Managing Schema Evolution

1. Understanding Schema Compatibility

Compatibility Type Description Notes
BACKWARD Consumer can read data produced by older version of schema Most common for evolving schemas
FORWARD Producer can read data produced by newer version of schema Less common, typically for consumers that don't process data
FULL Both BACKWARD and FORWARD compatibility Most restrictive, ensures full schema evolution support
NONE No compatibility between schemas Requires strict schema version matching, not recommended for evolving systems

Example: Schema Registry Configuration

{
  "schema.compatibility": "BACKWARD",
  "schema.registry.url": "http://localhost:8081"
}
Note: Kafka Schema Registry supports BACKWARD compatibility by default, allowing new schema versions to be compatible with older data.

Schema Compatibility Rules

  • Adding a new field with default value is BACKWARD compatible
  • Removing a field is NOT BACKWARD compatible
  • Changing a field type to a compatible one is BACKWARD compatible (e.g., int to long)
  • Changing a field type to an incompatible one is NOT BACKWARD compatible (e.g., string to int)

2. Implementing Backward Compatibility

Compatibility Type Description Examples Notes
Backward Compatible Schema can read data written by older versions @Schema(version = 1) with @Schema(version = 2) fields added Ensures older consumers can still read new data
Forward Compatible Schema can write data that older versions can read @Schema(version = 2) with @Schema(version = 1) fields removed Allows newer producers to write data for older consumers
Full Compatibility Both backward and forward compatible @Schema(version = 1) to @Schema(version = 3) with no field removals Ensures maximum interoperability across versions

Example: Adding Fields to Schema

@Schema(name = "User", version = 1)
public class UserV1 {
    private String name;
    private int age;
}

@Schema(name = "User", version = 2)
public class UserV2 {
    private String name;
    private int age;
    private String email; // New field added
}
Note: When adding new fields to a schema, ensure they are optional or have default values to maintain backward compatibility.
Warning: Removing fields or changing field types can break backward compatibility and should be avoided in schema evolution.

3. Implementing Forward Compatibility

Approach Description Implementation Notes
Schema Evolution Rules Forward compatibility ensures that new schemas can read data produced by older schemas. 
SchemaRegistry.register("topic-v1", schemaV1);
SchemaRegistry.register("topic-v2", schemaV2);
 Schema versions must be compatible in a forward direction.
Field Addition Adding new fields to a schema is forward compatible if they are optional or have default values. 
{ "name": "user_id", "type": "string" }
{ "name": "email", "type": "string", "default": "" }
 Fields without defaults must be optional in the consumer.
Field Removal Removing fields is not forward compatible unless they are no longer used in the data. 
{ "name": "user_id", "type": "string" }
{ "name": "email", "type": "string" }
 Old consumers will not understand removed fields.
Type Changes Changing field types is not forward compatible unless the new type can represent all values of the old type. 
{ "name": "user_id", "type": "string" }
{ "name": "user_id", "type": "int" }
 Schema validation will fail if type conversion is not possible.

Example: Schema Registry Configuration

{
  "schema.registry.url": "http://localhost:8081",
  "schema.compatibility": "FORWARD"
}
Note: Forward compatibility is enforced by the Schema Registry using compatibility rules.
Warning: Ensure all producers and consumers are updated to support new schema versions.

4. Using Full Compatibility

Compatibility Type Description Use Case Notes
BACKWARD_FULL Consumer can read data produced by newer schema versions Schema evolution where producers are updated before consumers Ensures backward compatibility with older versions
FORWARD_FULL Producer can write data using newer schema versions Schema evolution where consumers are updated before producers Ensures forward compatibility with newer versions
FULL Both backward and forward compatibility Schema evolution where both producers and consumers can be updated independently Most robust for managing schema changes in distributed systems

Example: Schema Evolution with FULL Compatibility

{
  "schema": "{\n  "type": "record",\n  "name": "UserEvent",\n  "fields": [\n    {\n      "name": "id",\n      "type": "string"\n    },\n    {\n      "name": "email",\n      "type": "string"\n    },\n    {\n      "name": "timestamp",\n      "type": "long"\n    }\n  ]\n}",
  "compatibility": "FULL"
}
Note: When using FULL compatibility, schemas must ensure that new fields are optional or have default values to maintain compatibility.

5. Using Transitive Compatibility

Transitive compatibility in Kafka Schema Registry allows schemas to evolve through a chain of backward and forward compatibility checks. When a new schema is registered, it must be compatible with all previous versions in the chain.

Schema Version Compatibility Type Description
1.0 BACKWARD Ensures new schema can read data produced by older schemas
2.0 TRANSITIVE_BACKWARD Ensures new schema can read data from all previous versions in the chain
3.0 TRANSITIVE_FORWARD Ensures older schemas can read data produced by the new schema

Example: Registering a Schema with Transitive Compatibility

curl -X POST http://localhost:8081/subjects/user-avro/versions \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  -d '{"schema": "{\n  \"type\": \"record\",\n  \"name\": \"User\",\n  \"fields\": [\n    {\"name\": \"id\", \"type\": \"int\"},\n    {\"name\": \"name\", \"type\": \"string\"}\n  ]\n}"}'
Note: Transitive compatibility requires that each schema in the version chain maintains strict backward and forward compatibility rules.

6. Configuring Compatibility Level

Compatibility Level Description Notes
BACKWARD Schema can read data written by previous versions Ensures backward compatibility with older schemas
FORWARD Schema can read data written by newer versions Ensures forward compatibility with newer schemas
FULL Schema is compatible in both directions (backward and forward) Most restrictive level, ensures full compatibility
BACKWARD_TRANSITIVE Schema can read data written by any previous version in the history Ensures compatibility with all past versions
FORWARD_TRANSITIVE Schema can read data written by any newer version in the history Ensures compatibility with all future versions
FULL_TRANSITIVE Schema is compatible in both directions with all past and future versions Most restrictive level, ensures full transitive compatibility

Example: Setting Compatibility Level in Schema Registry

{
  "compatibility": "BACKWARD"
}
Note: The compatibility level is configured at the subject (topic) level in Schema Registry.

7. Handling Schema Evolution Errors

Example: Schema Registry Error Handling

SchemaRegistryClient schemaRegistry = new SchemaRegistryClient();

try {
    Schema schema = schemaRegistry.getLatestSchemaMetadata(subject).getSchema();
} catch (SchemaRegistryException e) {
    if (e.getErrorCode() == 40401) { // Schema not found
        // Handle missing schema
        System.out.println("Schema not found for subject: " + subject);
    } else if (e.getErrorCode() == 409) { // Compatibility violation
        // Handle compatibility error
        System.out.println("Schema compatibility violated");
    }
}
Error Code Description Recommended Action
40401 Schema not found Recreate schema or use fallback
409 Compatibility violation Update consumer or adjust schema
400 Invalid schema Fix schema definition
Note: Kafka Schema Registry returns HTTP error codes for various schema evolution issues. Always implement retry logic and fallback mechanisms in production.
Warning: Ignoring schema evolution errors can lead to data corruption or consumer failures.

Common Schema Evolution Error Scenarios

  • Incompatible Changes: Adding required fields or changing field types
  • Missing Schema: Subject not found in registry
  • Version Mismatch: Consumer expecting different schema version

8. Versioning Schemas

Schema Version Description Usage Notes
Schema ID Unique identifier for a schema version Used in Avro serialization to reference the correct schema Assigned by Schema Registry
Schema Compatibility Defines how new schema versions can evolve from previous ones Configured at subject level in Schema Registry Controls backward/forward compatibility
Schema Evolution Types Backward, forward, full, and transitive compatibility modes Set via Schema Registry API or CLI Backward compatible schemas can read data written by older versions

Example: Schema Versioning in Avro

{
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "id",
      "type": "int"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "email",
      "type": "string",
      "default": ""
    }
  ]
}

Example: Adding a New Field to Schema

{
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "id",
      "type": "int"
    },
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "email",
      "type": "string",
      "default": ""
    },
    {
      "name": "phone",
      "type": "string",
      "default": ""
    }
  ]
}
Note: When a new field is added with a default value, it's considered backward compatible and can be read by older consumers.
Warning: Removing or changing the type of an existing field breaks backward compatibility.

9. Using Schema References

Schema references allow you to reuse and share schema definitions across multiple topics or schemas, promoting consistency and reducing redundancy.

Reference Type Description Usage
Schema ID Unique identifier for a schema in the registry Used to reference schemas when defining new ones or updating existing ones
Schema Name Name of the referenced schema Used for human-readable references in documentation or UIs

Example: Defining a Schema with References

{
  "type": "record",
  "name": "UserEvent",
  "fields": [
    {
      "name": "userId",
      "type": {
        "type": "string"
      }
    },
    {
      "name": "eventData",
      "type": {
        "type": "record",
        "name": "EventData",
        "fields": [
          {
            "name": "timestamp",
            "type": "long"
          },
          {
            "name": "action",
            "type": "string"
          }
        ]
      }
    }
  ]
}
Note: Schema references are supported in Confluent Schema Registry v7.0+ and require Avro 1.9+ for full compatibility.

10. Migrating Schema Formats

Format Description Migration Path Notes
Avro Schema format using JSON schema definition Use Schema Registry to convert and validate schemas Supports backward/forward compatibility rules
JSON Schema-less format using JSON objects Use JSON Schema or custom validation logic Less strict, more flexible but less safe
Protobuf Protocol Buffers with defined message types Use protobuf compiler to generate schemas Efficient binary format, good for performance

Example: Converting Avro to JSON Schema

// Avro schema definition
const avroSchema = {
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "id",
      "type": "int"
    },
    {
      "name": "name",
      "type": "string"
    }
  ]
};

// Convert to JSON Schema (simplified)
const jsonSchema = {
  "type": "object",
  "properties": {
    "id": { "type": "integer" },
    "name": { "type": "string" }
  },
  "required": ["id", "name"]
};

// Validate using JSON Schema library
const Ajv = require('ajv');
const ajv = new Ajv();
const validate = ajv.compile(jsonSchema);
const valid = validate({ id: 1, name: "John" });
Note: When migrating, ensure that all consumers can handle the new schema format to avoid data loss or processing errors.

Was this article helpful?

Related Cheat Sheets