Skip to content

feat(format): schema evolution for the Java row codec - #3714

Draft
stevenschlansker wants to merge 44 commits into
apache:mainfrom
stevenschlansker:row-codec-schema-versions
Draft

stevenschlansker wants to merge 44 commits into
apache:mainfrom
stevenschlansker:row-codec-schema-versions

Conversation

@stevenschlansker

@stevenschlansker stevenschlansker commented May 29, 2026 •

Copy link
Copy Markdown
Contributor

Opt in with .withSchemaEvolution() on any row, array, or map codec builder. Live fields added after the first version carry @ForyRowVersion(since = N); removed fields are declared with @ForyRowVersion(until = N) on a class referenced from
@ForyRowSchema(removedFields = ...). Older payloads are dispatched at read time. Standard and compact formats supported.

With the flag off, the wire format and the generated codecs are unchanged. Two changes apply regardless of the flag: TypeUtils.isSupported now passes the type-resolution context to map keys and values, so interface beans work as map keys and values in every row codec, and fory-format compiles with <release>11</release>.

Why?

Currently changing row format schema definition in any way invalidates all records

What does this PR do?

Propose a new concept of format versions, each succeeding version may add or remove fields from types, and deserialization machinery picks version based on schema hash

  Schema evolution for the Java row codec

  Lets a consumer built from the current bean decode rows written by older versions of that bean.

  Adds opt-in schema evolution to the Java row codec, enabled with
  `withSchemaEvolution()` on the bean/array/map codec builders. A reader can decode
  payloads written against older versions of a bean by dispatching on a per-payload
  strict schema hash to a projection codec that reads the historical layout and
  maps it onto the current type, applying defaults for fields that did not yet
  exist and discarding fields that have been removed.

  Field history is declared with `@ForyRowVersion(since)` on live fields and with
  `@ForyRowVersion(since/until)` on a `@ForyRowSchema(removedFields = ...)` history class for removed fields.
  `SchemaHistory` enumerates the version history, including the cross-product over
  nested versioned beans; each historical layout gets a projection codec whose row
  layout is precomputed once so projection decode costs the same per call as
  current-schema decode. The wire format uses an 8-byte strict-hash slot
  (reusing the row's existing hash slot, and a new prefix on every evolution-enabled array/map),
  and producer/consumer must agree on the flag.

  Covers the standard and compact row formats, records (including
  `@ForyRowVersion` on record components) and interface beans, and nested versioned
  beans to arbitrary depth.


  API
  - `@ForyRowVersion` — version window of a field: `since` on a live field, record component, or accessor method; `since`/`until` on a removed-field declaration. A live member that sets `until` fails the codec build.
  - `@ForyRowSchema` — optional; names the class that declares removed fields.

  How it works

  Row payloads already carry an 8-byte schema hash. Evolution-enabled encoders keep
  a map of historical hashes; on decode, a hash mismatch is resolved against that
  map instead of failing.

  - `SchemaHistory` derives the ordered historical schemas from the version
    annotations and a strict hash per version (hash includes field name and
    nullability).
  - `RowCodecBuilder` indexes one projection per historical combination of outer
    and nested versions. Its class is generated on the first decode of that
    combination's hash, reads the historical schema, and produces current-bean
    instances.
  - `BinaryRowEncoder.decode` dispatches on the peer hash: exact match → current
    codec; known historical hash → projection codec; else
    `ClassNotCompatibleException`.
  - Nested versioned beans have no hash of their own: their layouts fold into the
    enclosing payload's hash, and each combination's projection embeds the matching
    nested projections, so versioning composes through `List<Bean>` and `Map<_, Bean>`.

  Performance

  Both decode paths reuse the `CompactRowLayout` cache from #3717. The
  current-schema path allocates rows via `writer.newRow()`; the historical path via
  a per-projection `RowFactory` that holds its historical schema's layout, built
  once. Per-decode cost is the same on both — no layout recomputation. The
  evolution-enabled array/map codecs keep a single allocation per encode.

AI Contribution Checklist

AI Usage Disclosure

  • substantial_ai_assistance: yes
  • scope: all
  • affected_files_or_subsystems: row format Java
  • ai_review: <line-by-line self-review completed; summarize the two-reviewer loop and final no-further-comments result>
  • ai_review_artifacts:
  • human_verification: <checks run locally or in CI + pass/fail summary + contributor reviewed results>
  • performance_verification: ✔️
  • provenance_license_confirmation: ✔️
  • If yes, I included a completed AI Contribution Checklist in this PR description and the required AI Usage Disclosure.
  • If yes, my PR description includes the required ai_review summary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.

Does this PR introduce any user-facing change?

New codec option: schema evolution. Some small annotations and a builder method.
Existing row format compatibility unchanged

The protocol (version model, wire names, field order, strict hash, and framing) is specified language-neutrally
in docs/specification/row_format_spec.md. Java is the only implementation in this PR.

Benchmark

withSchemaEvolution() is an opt-in feature that adds a new row-codec path; it does not modify any existing serialization hot path.
There is no apache/main baseline to compare against — SchemaEvolutionSuite exercises withSchemaEvolution(), which does not exist
on main — so the benchmark measures two things directly: the steady-state cost of enabling the flag, and projection-vs-current
parity.

Bounded JMH run (JDK 26, 2 forks × 4 iterations, 1s each, -prof gc). B/op is gc.alloc.rate.norm (bytes allocated per operation).

Benchmark Throughput (ops/s) B/op
currentDecode 17.6M 312
currentDecodeNoEvolution 16.6M 312
encode 15.9M 152
encodeNoEvolution 15.8M 152
compactCurrentDecode 16.4M 280
compactCurrentDecodeNoEvolution 16.3M 280
compactEncode 16.4M 144
compactEncodeNoEvolution 15.5M 144
olderDecode 24.5M 216
compactOlderDecode 24.9M 192

Findings:

  • Enabling evolution adds zero allocation on the current path. B/op is byte-identical between the evolution-on and *NoEvolution
    variants on every path (decode 312/312, encode 152/152, compact decode 280/280, compact encode 144/144).
  • Throughput overhead of the flag is within the run's noise band. Every on-vs-off pair overlaps within error. This bounded run has
    ~10% confidence intervals on the no-evolution variants, so the throughput claim is "no measurable difference," not a tight bound;
    allocation is exact.
  • Projection (older-version) decode is not penalized versus current decode. It allocates less here (216 vs 312 B/op standard; 192
    vs 280 compact) because it reads the narrower V1 schema, not because projection is inherently cheaper. Each projection codec holds its
    historical schema's precomputed row layout, so there is no per-decode rebuild.

Limitations

  • Producer and consumer must agree on the withSchemaEvolution() flag; the two
    framings are not wire-compatible. A flag-mismatched peer fails loudly with
    ClassNotCompatibleException (except evolution-off reading evolution-on bytes,
    which is undefined). Turning the flag on is a cutover, not a rolling change:
    stop or drain producers, rewrite or stop reading stored payloads, then start
    every peer with the flag on in a release that changes no schema.
  • A reader decodes its own and older versions only. Deploy every reader of a
    schema change before any writer.
  • Java is the only implementation. Other runtimes (Python, C++, and so on) cannot
    read evolution-enabled payloads until they implement the protocol in the row
    format specification.
  • Building a codec computes the layout and hash of every combination of nested
    versions, so build time and memory grow as the product of the version counts of
    the distinct nested versioned bean classes. Projection classes generate lazily,
    on the first decode of each historical hash. To bound the growth, remove History
    entries or remove or lower a live field's since; the reader then rejects the
    payloads with the retired layouts.
  • Wire names use Fory's existing xlang snake_case conversion, which the runtimes do
    not yet agree on for acronyms, a leading capital, or non-ASCII capitals (xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree #4187).
    Ordinary lowerCamelCase names are portable.

Comment thread docs/row-format/java.md
Comment thread docs/guide/java/row-format.md Outdated
Comment thread docs/guide/java/row-format.md Outdated
Comment thread docs/row-format/java.md
@stevenschlansker
stevenschlansker force-pushed the row-codec-schema-versions branch 16 times, most recently from 7823b91 to 8be8335 Compare June 30, 2026 15:56
@stevenschlansker stevenschlansker changed the title Draft: feat(format): schema evolution for the Java row codec feat(format): schema evolution for the Java row codec Jun 30, 2026
@stevenschlansker
stevenschlansker force-pushed the row-codec-schema-versions branch 4 times, most recently from 5d7e5e3 to 4964041 Compare June 30, 2026 21:01
@stevenschlansker

stevenschlansker commented Jun 30, 2026 •

Copy link
Copy Markdown
Contributor Author

@chaokunyang , this is not 100% done yet, but I think it is getting very close. Would you mind doing a design review to make sure that this feature will be acceptable to commit into the project? It is not a small feature, but I think the complexity is justified by giving the row format a low-cost method of achieving schema evolution.

If there is some way to streamline the eventual code review of a large volume of LLM-typed code, please let me know.
I have tried to keep the bar for quality high (many, many review iterations per ai policy) and testing thorough.
I considered breaking it up into multiple PRs but there is not a "slice" of the feature that stands on its own.

I intend to test this over the coming month or so internally, and mark the PR as ready once we have some real world experience running it.

…lution walk

A codec supplying its own terminal foryField replaces the whole column,
but Wrapper.classify only recognized codecs that encode to a different
type, so a terminal-codec field fell through to the bean descend and
SchemaHistory introspected the declared type — failing the canonical-
order check for classes like libphonenumber's PhoneNumber whose member
names sort differently by wire name. Classify such fields as raw-less
leaves; intercepting codecs (encodedType == declared) still descend the
declared bean.
@stevenschlansker
stevenschlansker force-pushed the row-codec-schema-versions branch from 2e02e6a to c15698a Compare October 15, 2026 20:33
Claude (on behalf of Steven Schlansker) added 20 commits October 16, 2026 15:06
…ds reach

A removed field whose type is a versioned bean put that bean into the
outer combination's nested routing. Routing then compiled the bean's
historical projection, which referenced current-version codecs of its own
nested beans that no live field had caused to be generated, so the first
decode of an old payload failed to compile. Projection skips removed
slots, so those beans need no codec: route only the classes a live field
reaches.
The compact writer sorts every struct it writes, including list elements
and map keys and values, but the history transform sorted only the top
row and its direct struct children. A struct reached through a list or
map was hashed in wire-name order when its bean was unversioned and in
alignment order when it was versioned, so the hash did not describe the
bytes and another runtime could not reproduce it from the spec.

Sort every struct at every depth before hashing. This changes the strict
hash of compact layouts that hold a struct inside a list or map.
A map position that reached no bean contributed the constant 0 to the
combined map hash, so changing its type (Map<String, Bean> to
Map<Long, Bean>) kept the hash and decoded silently into wrong values.
The key position was also hashed as nullable although the map writes
keys as non-nullable. Hash both positions from their element history,
with the nullability the map writes them with. This changes the strict
hash of every evolution-enabled map.

The array and map projection sources were inner classes that read the
builder's mutable codecFormat at the first decode, and the map source
mixed it with a format argument from the encoder. Make them static and
capture the format, map type, and field at build time, as the row source
already does, and drop the redundant format argument.
A projection assigned only the fields the older payload carried, so on a
plain bean an absent field kept its field initializer, and an absent
Optional with no initializer stayed null. Records and interface beans
already got null, zero, false, or Optional.empty(). Assign the same
defaults to the absent fields of plain beans.
Row decode checked size < 8 before it read the schema hash, on the
flag-off path too, and the evolution array and map decoders did the
same. A short payload already fails on the hash comparison or on the
buffer bounds check, so the checks only changed the exception type.
Remove them and the tests that pinned that type.
- Pin the shared code generator only when a codec takes the evolution
  path.
- Give key-position bean codecs their own registration only in a map
  projection codec. A current-schema map codec reads both positions with
  the same codec, so Map<B, B> again embeds one B codec.
- Unwrap Optional in the evolution walk only when no codec is registered
  on Optional itself, as type inference does.
- Correct the withSchemaEvolution, ForyVersion.since, and benchmark
  comments.
Every existing hash test compared two computed hashes, so a refactor
that changed the hash itself still passed and would have broken every
payload already written. Pin scalar, nullable, UTF-8 name, decimal
(including a negative scale), binary, list, map, nested-struct, and
combined map hashes to values computed from the spec independently of
the Java implementation.
Specify that the strict hash covers fields in wire order at every depth,
which array and map positions gain a prefix, how each position's
one-field layout is built, and how a negative decimal scale mixes. Add
known-answer vectors. State the absent-field defaults that projection now
applies on every bean kind, and stop naming one exception type for a
flag mismatch.
… versions

A class reached both directly and through another versioned bean was a
separate dimension of the outer cross-product from its occurrence inside
that bean's history, so enumeration also built and hashed combinations
with the class at two versions. A writer has one definition of each
class, so those cannot occur on the wire. Keep only combinations that
fix every class at one version, at any depth. Infer each field once per
version rather than once per combination, and correct the guide, which
said the build cost tracked only the versions received.
…rict hash

An evolution-enabled array or map wrote the hash prefix only when its
element, key, or value reached a bean. The spec could not state that rule
across languages, and a payload with no bean, such as Map<String, Long>,
carried no hash: a reader built for Map<String, Integer> decoded it
silently into wrong values. Write the prefix whenever the flag is on, so
every payload written with evolution starts with the strict hash of its
layout. The reachable bean now only names generated projection classes.
…coder

The projection cache was per encoder, so every encoder's first decode of
an older hash rebuilt the codec builders, took the global code-generator
lock to look up the class, resolved the constructor handle, and built the
row layout. The guide recommends one encoder per virtual thread, so a
rolling upgrade repeated that work on every thread. Resolve the class,
constructor handle, and historical fields or row factory once on the
shared projection source, and keep only the writer binding per encoder.
…ss suffix maps

Array and map codec builders could still embed one uniform suffix on
every nested bean when no per-class map was set, a fallback no caller
reached any more. A later caller that reached it would have routed every
nested class to one class's projection. Keep the class suffix for naming
the generated array or map class only, route nested beans only through
the per-class maps, and remove the constructors that existed for the
fallback.
…two versions

The top-level map codec enumerated every (key version, value version)
pair, also when the key and value reach the same bean class. A writer
has one definition of each class, so a pair with the class at two
versions cannot occur on the wire. Skip such pairs, as the row
enumeration already does. The tests that wrote such pairs through two
stand-in classes now use distinct key and value classes, and one test
checks that a same-class pair at two versions is rejected.
State that the row wire name follows the xlang field-name rule, which
does not yet define acronyms, a leading capital, or non-ASCII capitals,
and that only ordinary lowerCamelCase names are portable today. Warn
that a history method with the wrong name changes the historical hash
and only fails when an old payload is decoded.
A fixed-width hash token ends at a fixed place, so the class-name suffix
shows its own token boundaries.
A row projection codec took the encoder's own writer as its root, so its
nested row and array writers registered as children of that writer.
Each projected version added writers that every later encode walked in
setBuffer, and kept them reachable for the encoder's lifetime. Give the
decode-only codec a detached writer, as the array and map projections
already have.
…alves

The value and key halves of a map projection class suffix were joined
with a _K marker. A value-side bean whose simple name repeated that
marker could produce the same class name as a different key-side
combination, and the shared code generator would then reuse the wrong
class. Write each half after its length so the join is injective, and
share the join with the test instead of copying it.
…tors

The hash vectors covered the hash function but not how the encoders
build the prefix input: the empty element name, the map key and value
nullability, the combine order, and the deep compact sort. Add framing
vectors for a map, an array, and a compact array to the spec, computed
independently of the Java implementation, and check the encoders
against them. Reword the spec sentence about evolution-off readers.
Add a test in which encoders from one row, array, and map codec decode
the same older hash for the first time at once, since those encoders
share each projection's lazily resolved state. Restate test comments
that narrated past bugs as the invariant under test.
Claude (on behalf of Steven Schlansker) added 7 commits October 16, 2026 16:48
The message that every row codec throws on a hash mismatch had changed
from decimal to hex hashes, also for codecs without schema evolution.
Restore the decimal format so that log matches that depend on it keep
working.
The race test shared its beans with other tests, so the projection
classes could already exist when it ran, and the race then covered only
the shared resolution, not the first compile. Give it its own outer and
nested versioned beans.
…ails

The spec said that every evolution payload starts with the strict hash
of its layout, but a map's prefix is the combined key and value hash.
Label the map vector's nullability, and use struct type rather than
class in the version model. State in ForyVersion.until that only
removed-field declarations may set it. Drop the code-generation timing
from the guide and state that build cost has no limit.
…and ForyRowSchema

ForyVersion and ForySchema sat beside core's ForyField and ForyStruct in
source and read as object-serialization features, but they affect only
row codecs built with withSchemaEvolution(). Keep the Fory prefix and
the fory-format annotation package, and name the row scope.
… layout

A field removed at one version and added again with the same name and
type gives both versions the same layout, so they collapse into one
history entry. The kept entry depended on the nested combination: with
the nested beans current it treated the field as live, otherwise as
removed, so the same payload lost the field's value when a nested bean
was at an older version. Merge the live fields and live nested classes
of every collapsed version into the kept entry.
…by layout

Each encoder sized its projection cache for the whole cross-product,
though most decode one or two older hashes; start it small. Add the
outer layout's strict hash to projection class names so that a name
follows the layout it decodes. Print array and map mismatch hashes in
decimal, as the row message does.
…nd size framing

Turning the flag on cannot be a rolling change: flag-on and flag-off
peers reject each other's rows, a flag-off reader decodes flag-on arrays
and maps incorrectly, and rows written before the cutover are not
readable after it. Say so, and say that readers of a schema change must
deploy before writers, since a reader rejects payloads from a newer
writer. Correct the advice on limiting build growth, which applied only
to removed fields. Specify where the hash sits relative to an embedded
payload size, and name the map prefix as the combined hash.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants