Meta deploys LLM-powered Just-in-Time Catching Tests at full scale, catching 74 serious bugs in 6 months

The approach

Meta Engineering published on 29 October 2026 the deployment results of Catching JiTTests — a system that uses LLMs to generate executable tests on the fly for every code change, catching regressions before they land. The approach is fundamentally different from AI-based code review (where an LLM reviews a diff for bugs): Catching JiTTests generates tests that actually execute, demonstrating concrete behavioral differences introduced by a change.The entire system runs overnight on spare capacity, targeting diffs submitted the previous day that are deemed 'risky' using a mechanism similar to Meta's Diff Risk Score and trained on previous code changes. It uses two parallel workflows: a 'dodgy diff' workflow that treats each diff as if it were a mutant and uses mutation-guided LLM test generation to expose behavioral differences, and an 'intent-aware' workflow that uses an LLM to infer a diff's actual intention, describes the risks inherent in implementing that intent, and generates tests targeting those failure points.

Filtering false positives

The system uses a multi-stage assessment pipeline to separate strong catches from false positives. An LLM-as-Judge ensemble — multiple state-of-the-art LLMs that classify each weak catch — is complemented by a rule-based assessor called RubFake (Rule-Based False-Positive Killing Environment) that provides verified, deterministic judgments. Together, they produce an overall ranking of catch candidates.

Engineers are only contacted for the highest-ranked candidates through a human-on-the-loop approach rather than human-in-the-loop — in other words, the system operates autonomously except for escalated findings, with engineers reviewing outcomes at an aggregate level rather than supervising every output. This distinction between 'on-the-loop' and 'in-the-loop' is a deliberate design choice to keep the system scalable.

The results

As of the publication date, 589 engineers at Meta (64 percent of the engineers reached by the system) have engaged with Catching JiTTests results and reported that they surface genuinely actionable findings, with a 60 percent rate of true positives. In the first half of 2026, the system played a key role in catching 74 serious (SEV) privacy bugs at the diff stage, before they landed in production, lowering the overall occurrence of privacy SEVs by 20 percent during that period.

The paper 'Just-in-Time Catching Test Generation at Meta' was presented at FSE 2026. The deployment is now no longer a research prototype: it is part of Meta's infrastructure.

What it means

The significance of the 60 percent true-positive rate at 589 engaged engineers is that it validates a design choice — using executable tests that demonstrate real behavioral differences rather than LLM-based code review opinions — as a production-viable approach at Meta's scale. The 20 percent drop in privacy SEVs is a concrete, measured outcome.

For the broader industry, Meta's deployment shows that the cost and complexity of automated test generation through LLMs is now low enough to run on every diff, overnight, across a massive codebase. That is a meaningful milestone for AI-assisted software engineering, and it is likely to accelerate adoption of similar approaches at other large tech companies.

Caveats and what to watch

The results are Meta-specific and depend on its engineering culture, codebase structure and available compute. Not every organization will see the same true-positive rate or SEV impact. The 'dodgy diff' approach is resource-intensive — overnight spare capacity is a requirement, not an option.

Watch whether Meta publishes the full FSE 2026 paper with methodologies and reproducibility details. Watch whether other large engineering organizations (Google, Microsoft, Apple) adopt or adapt the JiTTesting approach. Watch whether the system's bug-finding capability extends beyond privacy into security vulnerabilities.

Architecture details

The JiTTesting pipeline runs through five stages. Stage 1 (Targeting and Triggering) runs overnight on spare capacity, selecting diffs submitted the previous day that pass a risk filter. Stage 2 runs both the dodgy-diff and intent-aware workflows in parallel, generating candidate tests. Stage 3 runs an LLM-as-Judge ensemble where multiple state-of-the-art LLMs classify each weak catch as true or false positive.

Stage 4 complements the LLM ensemble with RubFake, a rule-based assessor that provides verified deterministic judgments the LLMs cannot match. Stage 5 uses the combined ranking to decide whether to reach out to human engineers — a deliberate 'human-on-the-loop' architecture that keeps the system scalable while ensuring engineers see only the highest-signal findings. This is contrasted with 'human-in-the-loop' approaches where every output gets human review.

The privacy focus

The 20% reduction in privacy SEVs is one of the most concrete outcome metrics published for an LLM-based testing system. Privacy bugs are notoriously hard to catch with traditional testing because they often involve configurations or edge conditions that static test suites do not cover — a regression that exposes user data to the wrong audience, for example, or a permission check that was accidentally lifted.

The fact that 74 such bugs were caught at the diff stage, before they ever reached production, is a strong argument for the approach: generating tests that demonstrate concrete behavioral differences catches classes of bugs that code review or static analysis would miss. The privacy focus also suggests Meta prioritized this system for the highest-risk category first, with potential expansion to security and reliability domains.