AMI Labs’ H-JEPA plans with a stack of JEPA world models

What paper landed, and who is on it?

arXiv lists 2610.06805 as submitted 5 October 2026. Authors are Wancong Zhang, Basile Terver, Michael Rabbat, Yann LeCun, and Randall Balestriero, with affiliations spanning NYU, AMI Labs, INRIA Paris, and Brown University. The project page at h-jepa.com summarizes the method and hosts figures from the paper.

VentureBeat covered the work on 8 October 2026, which is how many readers will encounter it; the primary technical objects remain the arXiv abstract/PDF and the GitHub code release.

Hierarchical latent planning sits in a busy week for world models. For related lines see Meta FAIR’s RoboJEPA and Odyssey-3.

Equal-contribution and equal-advising marks on the author list (Zhang/Terver; LeCun/Balestriero) are worth noting for citation hygiene, but they do not change the technical claim. The project page at h-jepa.com mirrors the hierarchy story with summary figures for hierarchical predictive learning, hierarchical planning, and DROID offline results—useful orientation, still secondary to the arXiv PDF/HTML.

What does H-JEPA claim to change?

Flat action-conditioned JEPA world models predict and plan at one timescale or with multiple horizons inside one shared latent space. The authors argue that forces a single representation to serve both precise control and long-range progress, which is hard when useful control detail is unpredictable far ahead.

H-JEPA trains a stack of JEPAs end-to-end. Higher levels pool lower-level latents and predict over longer intervals in their own spaces, with SIGReg-style regularization against collapse. Planning is coarse-to-fine: the top level optimizes toward the goal; its predicted states become subgoals for the level below until the bottom level emits actions.

When environment factors evolve at separated timescales, the paper says higher levels can discard fast detail while keeping slower task-relevant state—for example, maze position without full leg pose.

Training is end-to-end across the stack: higher levels pool lower-level latents and predict over longer intervals in their own spaces, with regularization against representation collapse. Planning is explicitly coarse-to-fine rather than a single CEM pass in one latent. That design is the paper’s answer to why multi-horizon prediction inside one shared space struggles when control detail is unpredictable far ahead.

Where do the authors say hierarchy helps—and where not?

Evaluations cover FourRoom Distractors, Visual AntMaze, Push-T, OGBench Cube, and offline planning on DROID. Against flat LeWM and a shared-space hierarchical baseline (HWM), the authors report better success–compute frontiers on FourRoom, AntMaze, and Cube when adding levels up to three. The AntMaze headline in the abstract is success from 18% to 73% with less planner compute.

On Push-T, two-level H-JEPA matches or exceeds LeWM across tested budgets, but deeper models perform poorly—attributed to short episodes leaving less usable data for longer clips. Selective abstraction is strongest when timescales separate; manipulation datasets with smaller frequency gaps show little of that probe pattern.

DROID needs an inverse-dynamics term so representations keep the moving robot rather than the static background. That is a different failure mode than generative world-model video quality; for action-faithful robot video models see also DreamTrue.

The AntMaze 18%→73% line is the abstract’s headline navigation gain under reduced planner compute; FourRoom-with-distractors and OGBench Cube are the other sim suites where the authors say deeper stacks improve the success–compute frontier up to three levels. Push-T is the cautionary counterexample: short episodes starve longer-clip levels of usable data, so deeper hierarchy underperforms. On DROID, inverse-dynamics supervision is required so latents track the moving robot instead of the static background—offline Fréchet path fidelity, not warehouse task success.

What can you run today?

The kevinghst/H-JEPA repository is MIT-licensed and describes code for the planning results: model-depth and projected-cost comparisons on the simulated suites plus open-loop DROID planning at 5 fps. It builds on LeWM and stable-worldmodel. Installation notes call for Python 3.11 and CUDA-oriented PyTorch wheels.

The README we opened focuses on training and evaluation scripts rather than a one-click robot deployment. Pretrained weight URLs are not a substitute for reading which configs reproduce which figure.

Interactive neural rendering for agents is a separate stack; MirroS’s AgentGarten renderer still has code-world practice loops on its TODO list.

Expect a research training stack: Python 3.11, CUDA PyTorch, configs that reproduce planning figures, and dependencies on LeWM / stable-worldmodel lineage code. Budget time to match the exact depth and loss terms for the figure you care about; a default train script is not automatically the AntMaze 73% configuration. Language-conditioned goals remain future work in the authors’ framing.

What should readers not infer?

Author-reported AntMaze and DROID numbers are not third-party leaderboard certifications. Offline Fréchet path fidelity on DROID is not closed-loop task success in a warehouse.

LeCun’s AMI Labs affiliation makes the paper newsworthy; it does not by itself establish product readiness or a timeline for language-conditioned goals, which the authors only propose as future work.

More hierarchy is a hyperparameter with failure modes on short-horizon data—not a universal upgrade knob.

AMI Labs branding and LeCun’s involvement explain why the paper travels outside robotics Twitter; they do not convert sim planning curves into a product roadmap. Compare H-JEPA to flat JEPA planners and to generative robot video models on separate axes—latent planning sample efficiency versus pixel fidelity—rather than forcing a single “best world model” ranking.

What did we not test?

We did not train H-JEPA, run AntMaze planning, or compute DROID Fréchet scores. This article reports arXiv 2610.06805, the h-jepa.com summary, and the GitHub README we opened on 10 October 2026.