Kyutai's CADRE video autoencoders let world models react in under 230 ms

What Kyutai released

Kyutai, the Paris AI lab, has published a paper and open code for CADRE, a family of video autoencoders built for one job: letting a video world model react to your keyboard or controller fast enough to feel live. The paper, Rethinking Video Autoencoders for Interactive World Models, went up on arXiv on 21 October 2026. Its authors are Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick Pérez and David Picard, with the work done at Kyutai; Picard is also affiliated with École des Ponts and the BAIR lab at UC Berkeley.

Alongside the paper, Kyutai released the training and inference code under the MIT License, a set of model weights on Hugging Face and a public demo. The release includes seven autoencoders and world models for four kinds of scene: a robot manipulating objects, human hands doing tasks, a car driving, and a camera walking through a home. The headline claim is that the world models run at 20 frames per second on a single Nvidia H100 and respond to an action in under 230 milliseconds, the point at which people start to notice that their own movement looks late.

That makes CADRE a building block rather than a product. Most of the attention in world models goes to the big generator that dreams up the next frame. Kyutai's argument is that the part people usually take off the shelf, the autoencoder that squeezes video into a compact code and turns it back into pixels, is what decides whether a world model can be interactive at all.

Why latency is the problem

A video world model works on latents, compressed tokens that stand for chunks of video. The generator predicts the next latent from what came before and from the action the user just pressed, and the autoencoder's decoder turns that latent into frames you can see. Most modern video autoencoders, including the ones inside Wan, LTX and Cosmos, compress time as well as space: one latent covers several frames at once.

That is efficient for making a film clip, but it is a problem for anything interactive. If one latent covers eight frames, the model has to wait for all eight frames' worth of time before it can show the effect of your input. The paper writes the delay as the number of frames per latent divided by the frame rate, plus the time to generate, plus the time to decode. At eight frames per latent and 20 frames per second, the wait alone is 400 milliseconds, well past the roughly 230 milliseconds at which people notice their own movement is lagging, a threshold the authors take from a 2009 study by Shimada and colleagues.

Compressing less is not a clean fix either. An autoencoder that encodes every frame on its own keeps the wait short but produces far more tokens, which makes the generator slower and the video choppier. The paper frames the whole design problem as a trade between three things: how quickly the model responds, how many frames per second it can sustain, and how good the video looks.

How CADRE compresses video

CADRE's answer combines a frozen image encoder with a heavy, fast decoder. On the encoding side it reuses Meta's DINOv3 vision transformer, the 86 million parameter ViT-B/16, without training it. A single trained 3D convolution folds DINOv3's features over a small number of frames into a 64-channel latent. The released variants are named for that shape: CADRE-2×16×16 groups two frames and 16 by 16 pixel patches, while CADRE-2×32×32 compresses space twice as hard.

The decoder is where most of the trained parameters sit. It is a causal diffusion transformer with 24 blocks and a width of 1024, so it only looks at the past, and it is trained to produce frames in one or two denoising steps through a method called progressive self-distillation. Training happens in a single stage with no adversarial loss, which avoids the fiddly discriminator tuning many image decoders need. Each autoencoder has about 475 million trained parameters on top of the frozen encoder.

The autoencoders were trained on a mixture of 23 public video sources totalling about 76,000 hours of web, robot, driving and camera footage, for 150,000 steps at a batch size of 96 on 16 H100 GPUs. The world model on top is a 1.2 billion parameter causal diffusion transformer that generates video at 288 by 512 pixels and 20 frames per second. For testing, each model sees two seconds of real context and then generates 4.8 seconds of new video from the actions, scored on 2,048 held-out samples per environment.

The results against Wan, LTX-2 and Cosmos

The paper compares CADRE against autoencoders from Wan 2.2, LTX-2, Cosmos, the frame-by-frame DC-AE and a recipe called MIRA, all used inside the same world model so only the autoencoder changes. Measured end to end, CADRE-2×32×32 responds in 185 milliseconds at 23.4 frames per second, and CADRE-2×16×16 in 216 milliseconds at 17.3 frames per second. The Wan 2.2 setup takes 317 milliseconds, LTX-2 492 milliseconds and Cosmos 605 milliseconds. DC-AE is the fastest at 118 milliseconds because it does not compress time, and MIRA comes in at 175 milliseconds.

Across 48 comparisons against MIRA, Wan, Cosmos and LTX-2, the authors report that CADRE is at least as fast and more than 2 percent better in 42 of them. The biggest gap is in driving, where the 2×16×16 model scores a Fréchet video distance of 162.6, against 599.7 for Wan and 913.3 for LTX-2; lower is better. LTX-2 also nearly ignores actions in indoor scenes, with an action-response score of 0.04.

The wins are not universal. In robot manipulation Cosmos posts a better video distance, 39.5 against CADRE's 47.6, and Wan follows robot actions more faithfully, with an action-response rate of 0.85 against 0.45 for CADRE-2×32×32. On raw reconstruction, measured as PSNR, Wan's autoencoder reaches 35.02 against 25.12 and 22.86 for the two CADRE variants. CADRE does not copy pixels as precisely; it produces video that looks right and reacts in time, which is the trade the authors chose on purpose.

Four lessons from the paper

Beyond the models, the paper is useful for its four design findings. First, when compression is strong, a diffusion decoder gives sharper video than a decoder trained only to match pixels and perceptual features. Swapping in the diffusion decoder lowered video distance by 1.4 to 2.5 times compared with MIRA, and a second denoising step cut image distance by up to 23 percent.

Second, time should be compressed inside the autoencoder, not by grouping latents later in the generator. Patching eight frames together at the generator stage dropped the action-response score from 0.72 to 0.56. Third, camera motion decides how hard you can compress. Squeezing space 32 times multiplied video distance by 2.4 in robot scenes, and in navigation long latents made video distance up to 64 percent worse. Visible seams also appeared at chunk boundaries, with boundary error 1.6 to 1.8 times the normal level at eight frames per latent.

Fourth, coarse tokens hurt motion first. Before the picture looks blurry, the model starts to lose track of how things move and how they respond to actions. For anyone building a world model, the practical message is to measure action response directly, not only frame quality.

Where it falls short

The resolution is modest. At 288 by 512 pixels the world models are far from the 720p or 1080p output of film-style video generators, and the scenes are narrow by design: four environments, each trained on its own data, rather than one model that can dream up any place. The world models are research artefacts, not a general game engine.

The numbers also come from the authors' own benchmark, with their choice of baselines and metrics. They are clear about the cases where CADRE loses, but independent tests in other setups will matter more than the paper's 42 out of 48. And the speed figures assume a top-end data-centre GPU; nobody has shown this running on a gaming card yet.

The demo makes the cost visible. According to the repository, the public site serves all four environments on one H100 with two active users in total, and extra visitors wait in a queue. Kyutai measured about 14 frames per second per user with a 95th percentile input latency of 207 to 212 milliseconds, excluding network transit, when two people share the GPU, and 20 frames per second at 162 milliseconds for a single user.

Weights, licence and demo

The code is MIT licensed, but the weights are not fully open. The model cards on Hugging Face use the Creative Commons BY-NC 4.0 licence, which rules out commercial use, and because the encoder is DINOv3, Meta's DINOv3 licence applies too. The Hugging Face collection holds autoencoders for each compression setting, the 2×16×16 world models, action probes used for scoring, and four multi-resolution variants trained for 400,000 steps. Kyutai recommends cadre-2x16x16-multires as the default autoencoder.

Anyone can try the models at Kyutai's CADRE demo, steering each scene from the keyboard. MarkTechPost covered the release on 23 October, and the paper's HTML version has the full tables and training details.

Why it matters

Interactive world models are one of the busiest areas in AI this autumn, from Odyssey's foundation world model to the real-time audio-visual Helix World from Noiz and HKUST and Nvidia's expanded Cosmos simulation models. Robotics teams want them to train and test robot policies in simulation, game studios want them as playable worlds, and car makers want them to rehearse rare road situations.

CADRE is a reminder that latency is set by the whole pipeline, not just the generator. A strong video model on top of an autoencoder that bundles eight frames will always feel sluggish, however fast the GPU. It also fits Kyutai's run of open robotics and real-time work, after its Reki 2B robot model with short-term memory earlier this month. For researchers, the release gives a tested starting point and a clear recipe; for everyone else, it is a sign that responsive, steerable video worlds are getting closer to running on a single machine.