NVIDIA PivotOPD trains agents to recover from early mistakes

What problem does PivotOPD target?

The abstract says on-policy distillation gives dense teacher supervision on student trajectories, but in multi-turn settings an early wrong action changes later states so errors compound.

Preliminary experiments across three Qwen3 models (8B to 235B), the authors write, found that more than half of failed rollouts contain a pivotal mistake that typically occurs early, and that guiding only a few turns after that mistake can restore success.

That focus on early failure modes pairs with other recent agent-learning reports such as Memento 3’s reflective rulebooks and Xiaomi’s MiMo-V2.6 RL report.

Standard on-policy distillation, the authors argue, can cut overall failures while barely moving the subset of failures that follow a pivotal turn, because recovery actions stay at vanishingly small probability and never enter the sampled learning signal. PivotOPD is their answer to that coverage gap.

Affiliations on the project page span Princeton University, NVIDIA, and the University of Maryland, with corresponding authors marked. Internship footnotes are authorship context, not a product support channel.

How does the method work, on the authors’ account?

PivotOPD has the teacher name a gold action at each pivotal mistake and recovery actions for the next K turns. Preventive distillation uses reverse KL toward the gold action; recovery distillation uses forward KL on recovery actions the student rarely samples.

The NVIDIA project page describes a privileged self-teacher: the frozen student conditioned on a hint, so named actions become token-level targets in the student’s own style, then both distillation terms enter a PPO update.

Environment replay matters: later recovery turns start from states reached by executing recovery actions in a copied environment. That assumes replayable tasks—an explicit dependency on the page’s explanation.

Ablations on the project page vary the recovery budget K. They report that one recovery turn suffices on WebShop and search QA, while ALFWorld’s longer action chains prefer K=2. Those are validation-curve claims from the authors’ plots, not defaults you can assume for every domain.

Privileged teachers that see hints raise the usual distillation caveat: production students will not receive those hints at inference time. The method’s claim is that the distilled weights internalize prevention and recovery without the hint.

What results do the authors report?

Against 13 baselines on ALFWorld, WebShop, and search-based QA, PivotOPD is reported as strongest on average for Qwen3-1.7B and Qwen3-8B students, including +5.5% over the strongest baseline on ALFWorld with the 1.7B student.

Recovery: on 72 replayed pivotal mistakes, PivotOPD recovers 72.7% versus 20.3% for standard OPD and 8.3% for the base model, per the project page.

On software engineering, a Nemotron-3.5 student gains +3.2% resolve rate on SWE-Bench Verified (62.8% to 66.0%) versus +0.2% for standard OPD. Keep the benchmark primer at SWE-bench explained.

Teachers used in the main tables are large Qwen models per the paper’s captions; a self-distillation setting where Qwen3-8B teaches itself is also reported as still best on the three suites.

Self-distillation results, where Qwen3-8B teaches itself, are presented to argue that much of the gain comes from where the teacher intervenes rather than from raw teacher scale. Even so, the main tables still use larger teachers; read the caption before quoting a column.

What is not available yet?

The project page lists Code (coming soon). We found no maintained public training repository to clone on 10 October from that page.

Inference cost is described as unchanged after training because PivotOPD changes training only. Training cost, teacher calls, and environment replay (K recovery turns) are extra during learning—authors discuss K=1 or 2 depending on the suite.

SWE-Bench transfer in the write-up audits the final committed action with preventive distillation alone because episodes are long and containerized. Do not assume the full recovery term was active there.

Until code lands, the reproducible unit is the paper’s algorithm description and reported tables. Do not invent a GitHub URL or claim an NVIDIA NGC container from the project page’s “coming soon” label.

What should researchers verify before citing the gains?

Reproduce seed averages and teacher IDs from the paper tables rather than quoting only the homepage highlight chips.

Confirm whether your environment supports exact replay after a pivotal action; without that, recovery distillation’s setup does not transfer cleanly.

Separate author-reported ALFWorld success rates from production ticket-resolution metrics. We did not run either.

When comparing to standard OPD, match teacher identity, student size, and seed count from the paper’s tables. A single-run demo is not the same evidence class as the reported three-seed averages.

What did we not test?

We did not train PivotOPD, replay ALFWorld episodes, or evaluate SWE-Bench Verified. This article reports arXiv 2609.40285 and the NVIDIA project page only.