HuggingFace proposes Model Integration Contracts to catch Transformers regressions before they ship

The bug that started it

On July 30, 2026, HuggingFace's Transformers library enabled PyTorch's scaled dot-product attention (SDPA) for the T5 architecture as a default. A side effect: the UMT5 variant's decoder could attend to future tokens during training, effectively peeking at the answer it was supposed to predict. Nothing crashed. The regression stayed in main for almost two months.

When the HuggingFace team later replayed those commits, a small UMT5 model running on CPU flagged the anomaly on every single replay. That small model was the prototype for what became the Model Integration Contract (MIC) framework.

Why bugs reach main

Transformers now supports over 750 model types. Its test suite runs 88,000-plus CPU tests on main and more than 200,000 GPU tests in a daily run. Integration tests load real checkpoints and exercise the full API on real hardware, but running those on every pull request would require keeping hundreds of GPUs busy around the clock.

"By the time a scheduled run finds a regression, it may already be in main," the HuggingFace team wrote".

The solution was to shrink the model rather than the test suite. Build a fixture: a small model derived from the real checkpoint's configuration, with fewer and narrower layers but preserving every kind of layer the original has, plus its tokenizer and processor files unchanged. Publish it once, reuse it across PRs, and run fast integration tests against it on a cheap CPU runner.

How MICs work

A Model Integration Contract has three parts: a fixture (the shrunk model published once and pinned), a usage path (a runnable Markdown example of something people do with the model, such as generating text or running a training step), and baselines (the outputs recorded when running that example, reviewed by a human). The contract holds if running the usage path on the fixture still produces the baselines.

A contract covers an architecture and a usage path, not a single checkpoint. Compatible checkpoints can share a fixture while each keeps its own expected results. The contracts are published in a dedicated huggingface-mic collection on the Hub.

For example, the SmolLM contract generates from prompts such as "The capital of France is" and compares its token IDs with a recorded answer. It also compares training loss, checks that gradients are finite, and verifies that an optimizer step changes the weights. A fleet dataset on the Hub pins each MIC to a frozen revision along with the package versions used to record its baselines.

Shrinking models without losing coverage

The first recipe was simple: make dimensions smaller and keep fewer layers. That was not enough for mixed architectures where keeping two layers is no use if both use sliding attention and the full-attention layers disappear. The recipe now has four steps: shrink the configuration while preserving different layer kinds including mixtures of sliding and full attention or dense and MoE layers; keep preprocessing unchanged by copying tokenizer and processor files byte for byte; build and validate the weights with seeded random weights or by training on a small batch; then freeze and record.

The Llama 4 Scout shrink report shows the scale: 108.6 billion parameters become 44.7 million. The vision stack goes from 34 layers to 2, and the language stack from 48 to 4, keeping the checked layer kinds and MoE routing style.

Building MICs with agents

A CLI tool called tmic creates the contract and builds the fixture. For a new model, an LLM drafts the usage code and manifest, and CPU and GPU runs measure the candidate baselines. The LLM drafts but does not approve; changed expectations require human review.

HuggingFace used agents to build MICs across model families. When they hit an edge case, they fixed the recipe itself rather than the one fixture. The published models and remaining gaps are documented in the coverage space on the Hub.

Random vs overfit weights

With random weights, the Whisper fixture barely responded to changes in audio. It ran, but that made it a poor test of the audio path. So the team trained it on a few tones paired with words such as "red" and "green." They were not teaching it speech recognition, just making audio changes visible in its outputs.

In the weight-initialization study, the random Whisper fixtures caught 18 of 24 injected faults. The overfit fixtures caught all 24. HuggingFace keeps random weights when they pass the checks and overfits when better sensitivity is needed, though training takes seconds and adds one more step that can fail to converge.

Open questions and next steps

As of November 1, 2026, the MIC job in Transformers CI reports failures without blocking PRs. Whether it should block depends on measurements the team has not yet collected: how many real bugs it catches on PRs rather than in replays, how often it generates false alarms, and how much work baseline reviews require.

Coverage is still partial across model families and the APIs people use. When a regression slips through, the team can add the missing usage path and replay it on the broken and fixed versions. HuggingFace also wants to link original Hub model pages to their MICs, so a developer or an agent using a checkpoint could start from an example tested on that checkpoint, its environment, and its expected outputs.