Backward, Forward, and Full Compatibility With Concrete Events

Reference

“This event change is backward compatible” is often said without naming the reader and writer.

Compatibility becomes much easier when I write both versions and ask who must read whose data.

backward: new reader reads old writer data
forward:  old reader reads new writer data
full:     both directions work

These names can be implemented with different history policies in a schema registry, but the reader-writer direction is the stable starting point.

Version 1 of a payment event

Use an Avro-like record:

{
  "type": "record",
  "name": "PaymentCaptured",
  "fields": [
    {"name": "payment_id", "type": "string"},
    {"name": "amount_minor", "type": "long"}
  ]
}

One event is:

{"payment_id": "pay_17", "amount_minor": 1250}

Now a new requirement adds currency.

Adding a field needs a reader default

Version 2 is:

{
  "type": "record",
  "name": "PaymentCaptured",
  "fields": [
    {"name": "payment_id", "type": "string"},
    {"name": "amount_minor", "type": "long"},
    {"name": "currency", "type": "string", "default": "USD"}
  ]
}

Can the V2 reader consume old V1 data? Yes under Avro resolution because the reader supplies "USD" when the writer had no currency. This is backward compatibility.

Can the V1 reader consume V2 data? Avro record resolution ignores writer fields that are absent from the reader, so currency is skipped. This is forward compatibility for the structural schema.

Together the change is structurally full-compatible in this example.

The Apache Avro specification defines reader-writer resolution, including default values and ignored writer fields.

The default may be structurally valid and semantically false

If historical payments were not all USD, the default makes the schema check pass while corrupting meaning.

I ask:

Is USD a real historical fact?
Is it only a parsing fallback?
Can the consumer distinguish unknown from USD?

A safer addition may be a union with explicit unknown:

{"name": "currency", "type": ["null", "string"], "default": null}

Then downstream code must handle missing knowledge rather than receiving a fabricated currency.

Compatibility tools validate representability. They do not validate business truth.

Removing a field reverses the question

Suppose V3 removes amount_minor.

An old V2 reader expects that field. To read V3 writer data, it needs a reader default for the missing value. Without one, forward compatibility fails.

A new V3 reader can usually read old V2 data and ignore the extra writer field. That is backward compatible structurally, but the new application may no longer perform calculations that required the amount.

The direction table is:

Change	New reader reads old?	Old reader reads new?
add field with valid reader default	often yes	often yes if extras ignored
add required field without default	no	often yes
remove field old reader requires	often yes	no

“Often” is intentional. The exact answer belongs to the serialization format and registry rules.

Renaming looks like remove plus add

Changing payment_id to transaction_id produces two unrelated names unless the schema language has an alias mechanism and the reader uses it correctly.

Without an alias:

remove payment_id
add transaction_id

That can fail in both practical directions when the fields are required.

Even with a schema alias, generated code, JSON exports, SQL projections, monitoring fields, and consumers outside the registry may still depend on the old name. I search the wider contract surface before calling a rename compatible.

Type promotion is not arbitrary conversion

Some formats allow specific promotions, such as a writer's integer being read as a wider numeric type. This does not mean every “larger-looking” type is safe.

Changing:

amount_minor: integer cents

to:

amount: floating major units

is a semantic and unit change even if a library can convert the bytes. Consumers may multiply by 100, compare exact values, or use the field as an idempotency input.

I create a new field or event version when the meaning changes.

Compatibility against latest and compatibility against history differ

A registry may check the new schema only against the latest registered version or transitively against every relevant historical version.

The sequence can hide a break:

V1 has field A
V2 has A and B with default
V3 removes A

V3 may be compatible with V2 under one direction and rule while a long-lived V1 consumer still fails against V3 data.

Confluent's Schema Registry compatibility documentation distinguishes backward, forward, full, and their transitive forms. I record which history the platform checks instead of only the mode name.

Event compatibility includes operations

The schema registry does not know:

whether an old consumer is still deployed;
whether a replay contains five-year-old events;
whether a field participates in partitioning or identity;
whether null, missing, empty, and default have different meaning;
whether downstream exports use another format;
whether the event's business invariant changed.

I maintain consumer evidence: versions in production, oldest replayable data, and contract tests using real old payloads.

My compatibility test matrix

For every schema change I run:

new code reads old payload
old code reads new payload
new code reads new payload
replay code reads the oldest retained payload
unknown fields survive or disappear as intended
missing/default/null values retain business meaning
partition and identity fields remain stable

I include code generation and application assertions, not only the registry's schema check.

Backward, forward, and full compatibility are useful because they force direction into the conversation. They are not certificates that the product meaning survived. I start with reader versus writer, then test history, generated code, and the business interpretation of every fallback.