Repository navigation
feat(format): schema evolution for the Java row codec - #3714
stevenschlansker wants to merge 44 commits into
Conversation
7823b91 to
8be8335
Compare
5d7e5e3 to
4964041
Compare
|
@chaokunyang , this is not 100% done yet, but I think it is getting very close. Would you mind doing a design review to make sure that this feature will be acceptable to commit into the project? It is not a small feature, but I think the complexity is justified by giving the row format a low-cost method of achieving schema evolution. If there is some way to streamline the eventual code review of a large volume of LLM-typed code, please let me know. I intend to test this over the coming month or so internally, and mark the PR as ready once we have some real world experience running it. |
…lution walk A codec supplying its own terminal foryField replaces the whole column, but Wrapper.classify only recognized codecs that encode to a different type, so a terminal-codec field fell through to the bean descend and SchemaHistory introspected the declared type — failing the canonical- order check for classes like libphonenumber's PhoneNumber whose member names sort differently by wire name. Classify such fields as raw-less leaves; intercepting codecs (encodedType == declared) still descend the declared bean.
2e02e6a to
c15698a
Compare
… a Java-only implementation
…ds reach A removed field whose type is a versioned bean put that bean into the outer combination's nested routing. Routing then compiled the bean's historical projection, which referenced current-version codecs of its own nested beans that no live field had caused to be generated, so the first decode of an old payload failed to compile. Projection skips removed slots, so those beans need no codec: route only the classes a live field reaches.
The compact writer sorts every struct it writes, including list elements and map keys and values, but the history transform sorted only the top row and its direct struct children. A struct reached through a list or map was hashed in wire-name order when its bean was unversioned and in alignment order when it was versioned, so the hash did not describe the bytes and another runtime could not reproduce it from the spec. Sort every struct at every depth before hashing. This changes the strict hash of compact layouts that hold a struct inside a list or map.
A map position that reached no bean contributed the constant 0 to the combined map hash, so changing its type (Map<String, Bean> to Map<Long, Bean>) kept the hash and decoded silently into wrong values. The key position was also hashed as nullable although the map writes keys as non-nullable. Hash both positions from their element history, with the nullability the map writes them with. This changes the strict hash of every evolution-enabled map. The array and map projection sources were inner classes that read the builder's mutable codecFormat at the first decode, and the map source mixed it with a format argument from the encoder. Make them static and capture the format, map type, and field at build time, as the row source already does, and drop the redundant format argument.
A projection assigned only the fields the older payload carried, so on a plain bean an absent field kept its field initializer, and an absent Optional with no initializer stayed null. Records and interface beans already got null, zero, false, or Optional.empty(). Assign the same defaults to the absent fields of plain beans.
Row decode checked size < 8 before it read the schema hash, on the flag-off path too, and the evolution array and map decoders did the same. A short payload already fails on the hash comparison or on the buffer bounds check, so the checks only changed the exception type. Remove them and the tests that pinned that type.
- Pin the shared code generator only when a codec takes the evolution path. - Give key-position bean codecs their own registration only in a map projection codec. A current-schema map codec reads both positions with the same codec, so Map<B, B> again embeds one B codec. - Unwrap Optional in the evolution walk only when no codec is registered on Optional itself, as type inference does. - Correct the withSchemaEvolution, ForyVersion.since, and benchmark comments.
Every existing hash test compared two computed hashes, so a refactor that changed the hash itself still passed and would have broken every payload already written. Pin scalar, nullable, UTF-8 name, decimal (including a negative scale), binary, list, map, nested-struct, and combined map hashes to values computed from the spec independently of the Java implementation.
Specify that the strict hash covers fields in wire order at every depth, which array and map positions gain a prefix, how each position's one-field layout is built, and how a negative decimal scale mixes. Add known-answer vectors. State the absent-field defaults that projection now applies on every bean kind, and stop naming one exception type for a flag mismatch.
… versions A class reached both directly and through another versioned bean was a separate dimension of the outer cross-product from its occurrence inside that bean's history, so enumeration also built and hashed combinations with the class at two versions. A writer has one definition of each class, so those cannot occur on the wire. Keep only combinations that fix every class at one version, at any depth. Infer each field once per version rather than once per combination, and correct the guide, which said the build cost tracked only the versions received.
…rict hash An evolution-enabled array or map wrote the hash prefix only when its element, key, or value reached a bean. The spec could not state that rule across languages, and a payload with no bean, such as Map<String, Long>, carried no hash: a reader built for Map<String, Integer> decoded it silently into wrong values. Write the prefix whenever the flag is on, so every payload written with evolution starts with the strict hash of its layout. The reachable bean now only names generated projection classes.
…coder The projection cache was per encoder, so every encoder's first decode of an older hash rebuilt the codec builders, took the global code-generator lock to look up the class, resolved the constructor handle, and built the row layout. The guide recommends one encoder per virtual thread, so a rolling upgrade repeated that work on every thread. Resolve the class, constructor handle, and historical fields or row factory once on the shared projection source, and keep only the writer binding per encoder.
…ss suffix maps Array and map codec builders could still embed one uniform suffix on every nested bean when no per-class map was set, a fallback no caller reached any more. A later caller that reached it would have routed every nested class to one class's projection. Keep the class suffix for naming the generated array or map class only, route nested beans only through the per-class maps, and remove the constructors that existed for the fallback.
…two versions The top-level map codec enumerated every (key version, value version) pair, also when the key and value reach the same bean class. A writer has one definition of each class, so a pair with the class at two versions cannot occur on the wire. Skip such pairs, as the row enumeration already does. The tests that wrote such pairs through two stand-in classes now use distinct key and value classes, and one test checks that a same-class pair at two versions is rejected.
State that the row wire name follows the xlang field-name rule, which does not yet define acronyms, a leading capital, or non-ASCII capitals, and that only ordinary lowerCamelCase names are portable today. Warn that a history method with the wrong name changes the historical hash and only fails when an old payload is decoded.
A fixed-width hash token ends at a fixed place, so the class-name suffix shows its own token boundaries.
A row projection codec took the encoder's own writer as its root, so its nested row and array writers registered as children of that writer. Each projected version added writers that every later encode walked in setBuffer, and kept them reachable for the encoder's lifetime. Give the decode-only codec a detached writer, as the array and map projections already have.
…alves The value and key halves of a map projection class suffix were joined with a _K marker. A value-side bean whose simple name repeated that marker could produce the same class name as a different key-side combination, and the shared code generator would then reuse the wrong class. Write each half after its length so the join is injective, and share the join with the test instead of copying it.
…tors The hash vectors covered the hash function but not how the encoders build the prefix input: the empty element name, the map key and value nullability, the combine order, and the deep compact sort. Add framing vectors for a map, an array, and a compact array to the spec, computed independently of the Java implementation, and check the encoders against them. Reword the spec sentence about evolution-off readers.
Add a test in which encoders from one row, array, and map codec decode the same older hash for the first time at once, since those encoders share each projection's lazily resolved state. Restate test comments that narrated past bugs as the invariant under test.
The message that every row codec throws on a hash mismatch had changed from decimal to hex hashes, also for codecs without schema evolution. Restore the decimal format so that log matches that depend on it keep working.
The race test shared its beans with other tests, so the projection classes could already exist when it ran, and the race then covered only the shared resolution, not the first compile. Give it its own outer and nested versioned beans.
…ails The spec said that every evolution payload starts with the strict hash of its layout, but a map's prefix is the combined key and value hash. Label the map vector's nullability, and use struct type rather than class in the version model. State in ForyVersion.until that only removed-field declarations may set it. Drop the code-generation timing from the guide and state that build cost has no limit.
…and ForyRowSchema ForyVersion and ForySchema sat beside core's ForyField and ForyStruct in source and read as object-serialization features, but they affect only row codecs built with withSchemaEvolution(). Keep the Fory prefix and the fory-format annotation package, and name the row scope.
… layout A field removed at one version and added again with the same name and type gives both versions the same layout, so they collapse into one history entry. The kept entry depended on the nested combination: with the nested beans current it treated the field as live, otherwise as removed, so the same payload lost the field's value when a nested bean was at an older version. Merge the live fields and live nested classes of every collapsed version into the kept entry.
…by layout Each encoder sized its projection cache for the whole cross-product, though most decode one or two older hashes; start it small. Add the outer layout's strict hash to projection class names so that a name follows the layout it decodes. Print array and map mismatch hashes in decimal, as the row message does.
…nd size framing Turning the flag on cannot be a rolling change: flag-on and flag-off peers reject each other's rows, a flag-off reader decodes flag-on arrays and maps incorrectly, and rows written before the cutover are not readable after it. Say so, and say that readers of a schema change must deploy before writers, since a reader rejects payloads from a newer writer. Correct the advice on limiting build growth, which applied only to removed fields. Specify where the hash sits relative to an embedded payload size, and name the map prefix as the combined hash.
Opt in with
.withSchemaEvolution()on any row, array, or map codec builder. Live fields added after the first version carry@ForyRowVersion(since = N); removed fields are declared with@ForyRowVersion(until = N)on a class referenced from@ForyRowSchema(removedFields = ...). Older payloads are dispatched at read time. Standard and compact formats supported.With the flag off, the wire format and the generated codecs are unchanged. Two changes apply regardless of the flag:
TypeUtils.isSupportednow passes the type-resolution context to map keys and values, so interface beans work as map keys and values in every row codec, and fory-format compiles with<release>11</release>.Why?
Currently changing row format schema definition in any way invalidates all records
What does this PR do?
Propose a new concept of format versions, each succeeding version may add or remove fields from types, and deserialization machinery picks version based on schema hash
AI Contribution Checklist
AI Usage Disclosure
yes, I included a completed AI Contribution Checklist in this PR description and the requiredAI Usage Disclosure.yes, my PR description includes the requiredai_reviewsummary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.Does this PR introduce any user-facing change?
New codec option: schema evolution. Some small annotations and a builder method.
Existing row format compatibility unchanged
The protocol (version model, wire names, field order, strict hash, and framing) is specified language-neutrally
in
docs/specification/row_format_spec.md. Java is the only implementation in this PR.Benchmark
withSchemaEvolution()is an opt-in feature that adds a new row-codec path; it does not modify any existing serialization hot path.There is no
apache/mainbaseline to compare against —SchemaEvolutionSuiteexerciseswithSchemaEvolution(), which does not existon
main— so the benchmark measures two things directly: the steady-state cost of enabling the flag, and projection-vs-currentparity.
Bounded JMH run (JDK 26, 2 forks × 4 iterations, 1s each,
-prof gc).B/opisgc.alloc.rate.norm(bytes allocated per operation).currentDecodecurrentDecodeNoEvolutionencodeencodeNoEvolutioncompactCurrentDecodecompactCurrentDecodeNoEvolutioncompactEncodecompactEncodeNoEvolutionolderDecodecompactOlderDecodeFindings:
B/opis byte-identical between the evolution-on and*NoEvolutionvariants on every path (decode 312/312, encode 152/152, compact decode 280/280, compact encode 144/144).
~10% confidence intervals on the no-evolution variants, so the throughput claim is "no measurable difference," not a tight bound;
allocation is exact.
vs 280 compact) because it reads the narrower V1 schema, not because projection is inherently cheaper. Each projection codec holds its
historical schema's precomputed row layout, so there is no per-decode rebuild.
Limitations
withSchemaEvolution()flag; the twoframings are not wire-compatible. A flag-mismatched peer fails loudly with
ClassNotCompatibleException(except evolution-off reading evolution-on bytes,which is undefined). Turning the flag on is a cutover, not a rolling change:
stop or drain producers, rewrite or stop reading stored payloads, then start
every peer with the flag on in a release that changes no schema.
schema change before any writer.
read evolution-enabled payloads until they implement the protocol in the row
format specification.
versions, so build time and memory grow as the product of the version counts of
the distinct nested versioned bean classes. Projection classes generate lazily,
on the first decode of each historical hash. To bound the growth, remove
Historyentries or remove or lower a live field's
since; the reader then rejects thepayloads with the retired layouts.
not yet agree on for acronyms, a leading capital, or non-ASCII capitals (xlang: the snake_case conversion of field names is not specified, and the runtimes do not agree #4187).
Ordinary lowerCamelCase names are portable.