A Silent UMT5 Bug Lived in Transformers for Two Months. A Hugging Face Contract Test Runs in 13 Seconds on a CPU

Transformers enabled SDPA for T5 on July 30, 2026, and UMT5's decoder silently attended to future tokens during training for almost two months. Hugging Face caught it with a Model Integration Contract - a 66 MB fixture, a runnable usage path, and human-reviewed baselines - that a CPU can run in 13.4 seconds.

The bug that nobody noticed for two months

On July 30, 2026, Hugging Face turned on PyTorch scaled dot-product attention (SDPA) for T5 in Transformers. It was a performance change, and it was supposed to be invisible - invisible to every test in the suite, and invisible to the Hugging Face contract test that did not exist yet.

For UMT5 it was neither. As a side effect, UMT5’s decoder could attend to future tokens, so during training the model could peek at the answer it was supposed to predict. Nothing crashed, no test failed, no warning fired, and the regression stayed in Transformers main from July 30 until September 28, 2026 - almost two months - in a codebase that already runs 88,000+ CPU tests on main and 200,000+ GPU tests in a daily run.

The reason it survived is the reason most model bugs survive: the giant suite tests the code, and this was a behaviour of the code that was still correct enough to pass. Catching it took a different kind of test: a Hugging Face contract test.

What a Hugging Face contract test actually is

Hugging Face did not add another test to the giant suite. It built a new kind of artifact, and it published the standard for it on November 3, 2026: the Model Integration Contract (MIC).

A Model Integration Contract has three parts:

  • a fixture - a shrunk model built from the real checkpoint’s configuration, published once and pinned
  • a usage path - a runnable Markdown example of something people actually do with the model
  • a baselines record - the outputs recorded from running that example, reviewed by a human

The contract holds if running the usage path against the fixture still reproduces the baselines. That is the whole idea: pin the behaviour, not just the shape of the code.

Scale is what makes this practical. The UMT5 fixture carried 66 MB of weights against a 1,227 MB source checkpoint, and it ran on a cheap CPU runner. The selected T5 contract’s runner step took 13.4 seconds on CPU, excluding setup. That 66 MB figure belongs to the UMT5 fixture specifically - fixture size varies by architecture - but the order of magnitude is the point. A contract test you have to schedule on a GPU cluster is not a contract test anyone runs on every commit.

The replay found the exact window

Replaying the frozen contracts over five months of main flagged the regression from commit 560f36c (July 30, 2026) through the commit that fixed it, d7ba937 (September 28, 2026). Running the usage path on the fixture produced different numbers from the reviewed baseline: label log-probabilities differed by 1.22, training loss by about 0.20.

Two details generalise better than the headline number does.

First, the contract called SDPA, the default attention path. The eager-attention variant missed the bug entirely. A fixture only helps if it goes through the same code as the real checkpoint.

Second, the numbers are the finding. Nothing raised. Training loss moved, label log-probabilities moved, and every test in the suite was fine.

The honest part: one caught, four missed, one false alarm

Hugging Face published its own scorecard, and it is not flattering. Over five months of main, the fleet caught 1 regression, missed 4, and raised 1 false alarm. The misses were calls no contract made - the fleet was not wrong, it was blind.

Adding those missing calls caught all five with no new false alarm, and the new Whisper call found a live bug (huggingface/transformers#49403).

The pattern behind that number is the argument for the whole approach: of the 18 regressions the fleet caught, 6-7 raised an error and 11 returned a wrong value with no error at all.

The dangerous failure mode of a model dependency is not a crash. It is a plausible wrong answer. A crash is a cost. A wrong answer nobody notices gets trained into a checkpoint, gets evaluated against baselines that were themselves recorded from the buggy build, and ships.

A nightly run that caught a break the same night

A scheduled run of the contracts caught a regression the night it merged: PR #48282 (October 8, 2026) made generate() raise on every original Whisper checkpoint. The PR’s own CI could not see it - model tests build configs without suppress_tokens, pipeline tests pass their own generation_config, and the slow generate() tests were not in the run-slow list.

The same run found seven audio models leaving masked_spec_embed uninitialized (#49385, fixed in #49386).

For scale, the fleet dataset held 6 control contracts and 180 prototype contracts as of November 2, 2026, against 750+ supported model types. Those counts move, so read them as of that date.

What this does not mean

Two things this post does not claim.

It does not claim Transformers is covered. The project’s own evidence file says coverage is still partial: 4 of the 5 backfilled regressions were missed by the original fleet, and assisted decoding #48108 remains missed.

And it does not claim the UMT5 bug was a security problem. It was a training-time correctness bug in UMT5 - not a vulnerability, not a backdoor, and not an inference bug in shipped checkpoints.

The takeaway

A model in your stack is a dependency, and almost nobody contract-tests their dependencies. They unit-test the wrapper, they smoke-test the output, and they call it coverage.

Hugging Face’s answer was not more tests. It was a new artifact - a fixture, a usage path, and human-reviewed baselines - small enough to run on a CPU in 13.4 seconds, pinned, replayable over git history, and cheap enough to run on every commit. The evidence is public and replayable, which is the mitigation: anyone can check the claim.

That is the part worth stealing. When a dependency fails silently, the fix is not a louder crash. It is a smaller, pinned promise about what the dependency does, checked every time the code moves.

Sources

  1. Model Integration Contracts - Hugging Face blog, published November 3, 2026
  2. Evidence file for the transformers-mic contract fleet - Hugging Face, live repository file
  3. huggingface/transformers-mic - GitHub repository, live
  4. smollm-mic fixture - Hugging Face Hub, live
  5. transformers-fleet contract dataset - Hugging Face Hub, live
  6. Commit 560f36c - enable SDPA for T5 - GitHub, July 30, 2026
  7. Commit d7ba937 - the fix - GitHub, September 28, 2026

All figures above are Hugging Face reporting on Hugging Face. The evidence is public and replayable, which is the mitigation.