Avro Schema Evolution: Compatibility Guide

I've debugged enough 3 AM deserialization failures to know: schema evolution isn't optional. Your schemas will change. The question is whether those changes break consumers or not.

Schema Registry enforces compatibility rules. Understanding them is the difference between smooth deployments and production incidents. Visual schema management makes tracking versions and compatibility across all your clusters simpler.

We added a field without a default value. Deploy passed. Then 47 consumers started throwing deserialization errors. One line of JSON would have prevented it.

Data Engineer at an e-commerce company

The Four Compatibility Modes

Mode	Who Reads What	Upgrade Order
BACKWARD (default)	New consumers read old data	Consumers first
FORWARD	Old consumers read new data	Producers first
FULL	Both directions work	Any order
NONE	No checking	Careful coordination

BACKWARD is the default because you can always rewind consumers to replay historical data.

The Compatibility Matrix

Memorize this table:

Change	BACKWARD	FORWARD	FULL
Add field with default	✓	✓	✓
Add field without default	✗	✓	✗
Remove field with default	✓	✗	✗
Remove field without default	✓	✗	✗
Rename field	✗	✗	✗
Change field type	✗	✗	✗

Key insight: Adding a field without a default breaks BACKWARD compatibility. Old data doesn't have that field, and there's no default to use.

Safe Schema Evolution

Start with this schema:

{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "string"},
    {"name": "email", "type": "string"}
  ]
}

Adding a field (FULL compatible):

{"name": "country", "type": "string", "default": "US"}

New consumers use "US" for old data. Old consumers ignore the new field. Both directions work.

Adding a nullable field:

{"name": "phone", "type": ["null", "string"], "default": null}

The union type with default: null makes this explicitly optional.

Type changes never work. Changing int to string breaks everything. Use a new topic or migration strategy.

Test Before Deploying

This curl command prevents production incidents:

curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{...your new schema...}"}' \
  "http://localhost:8081/compatibility/subjects/users-value/versions/latest?verbose=true"
# {"is_compatible":true} or {"is_compatible":false,"messages":["...reason..."]}

Add ?verbose=true to see why a schema fails. Run this in CI before every deployment.

Handling Breaking Changes

Sometimes you need incompatible changes. Options:

New topic: Create users-v2 with the new schema. Migrate producers and consumers. Cleanest approach.

Disable compatibility temporarily:

# Set to NONE
curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "NONE"}' \
  http://localhost:8081/config/users-value

# Register breaking schema, then restore
curl -X PUT ... --data '{"compatibility": "BACKWARD"}'

This risks deserialization failures for any consumer not upgraded simultaneously.

Transitive Modes

Each mode has a transitive variant: BACKWARD_TRANSITIVE, FORWARD_TRANSITIVE, FULL_TRANSITIVE.

Non-transitive checks against the previous version only. Transitive checks against all versions.

Use transitive when: Consumers might replay from the beginning of the topic, or during disaster recovery.

Common Errors

"Schema being registered is incompatible" — Your change violates the compatibility mode. Add default values or change the mode.

"Reader missing default value" — You added a field without a default in BACKWARD mode. Add "default": "...".

"Writer field missing from reader" — You removed a required field in FORWARD mode. Add a default to the field first, then remove in the next version.

Best Practices

1. Use FULL compatibility when possible—safest mode
2. Always add defaults to new fields, even if you think you'll always have a value
3. Test in CI with the compatibility API
4. Avoid enums for frequently changing values—use strings instead
5. Use unions for optional fields: ["null", "string"]

Schema evolution is a contract between producers and consumers. Schema Registry enforces that contract. Get the compatibility mode right and you can evolve schemas without breaking production.

Book a demo to see how Conduktor Console provides visual schema management and compatibility testing across all your clusters.