Noiz AI and HKUST release Helix World, a real-time world model with stereo sound

What did Noiz AI and HKUST release?

Noiz AI, an audio-AI company, and HKUST published Meet Helix World on 12 October 2026. Helix World is a world model: it keeps generating an explorable scene while you steer it. The difference from most interactive world models is that it also generates the matching stereo soundtrack in the same pass, so when the camera turns, the sound turns with it.

The release has been staged over six weeks. Inference code and a preview checkpoint went up on 3 September, the technical report HelixWorld: A Real-time Interactive Audio-Visual World Model landed on arXiv on 29 September, and on 12 October the team published the launch post and HelixBench, the benchmark it uses to score sound that follows the camera. A browser demo runs at helixworld.org.

The demo accepts a starting image (or an upload), encodes it at 768×512, and takes an optional text prompt; leave it blank and the scene is described from the frame automatically. Generation runs in chunks of about 1.3 seconds, and each key press or new prompt takes effect from the next chunk. It is the sound-first counterpart to recent silent world models we covered, such as Odyssey-3 and MBZUAI’s WorldGuide.

The paper lists sixteen authors from HKUST and Noiz AI, with Lei Ke, Jiahao Pan and Zeyue Tian as equal contributors and Zeyue Tian and Wei Xue as corresponding authors. HKUST’s Qifeng Chen and Yike Guo are among the co-authors.

How fast is it, and how good is the picture?

The team scores picture quality and action following on the navigation split of WBench, and measures speed as real-time factor (RTF): seconds of compute per second of output video on one H800, including video and audio decoding. Below 1.0 is faster than real time.

Helix World scores 79.9 at an RTF of 0.77. Among models in the paper’s Table 3 that run faster than real time, that is the highest score: SANA-WM reaches 76.0 at 0.79 and Matrix-Game 3 reaches 71.3 at 0.58. Three models score higher but run slower than real time: Alaya-EVOKE-Turbo (82.0 at 4.29), EchoWM (81.0 at 1.98) and Zing-0.5 (81.0 at 2.81). Of those three, only EchoWM generates sound.

So Helix World does not win on visuals. Its claim is narrower: it is the only real-time model in that table that also produces audio. The LTX-2.3 base model it starts from scores 74.2 and needs 14.61 seconds of compute per second of video, so the speed comes from the distillation work described below, not from the base model.

Does the sound actually follow the camera?

Existing world-model benchmarks score video only, and audio-visual benchmarks use short clips without camera motion, the authors argue. HelixBench fills that gap with 1,015 human-verified cases across six domains and four subsets: Navigation, Perspective, Event and Subject. Each case provides a starting frame, an audio-visual prompt, continuous camera poses and a stereo reference recording.

Its spatial score correlates where a sound source sits on screen with left-right stereo panning, so it directly tests whether the sound field rotates with the camera. Helix World scores 41.8. EchoWM, the only other world model in the comparison that generates sound, scores 12.6 in the post’s table (the text above it says 12.7). Pipelines that mute Helix World’s video and re-dub it with a video-to-audio model score 14.8 (ThinkSound), −5.7 (AudioX) and −9.5 (PrismAudio). The negative numbers mean the sound pans the wrong way, because a dubbing model does not know where sources sit relative to the camera.

Helix World also leads on audio fidelity (KL 1.39, FAD 2.39) and on ImageBind audio-visual alignment. It is not best everywhere: LTX-2.3 (0.40 s) and the ThinkSound pipeline (0.42 s) have tighter timing on DeSync than Helix World’s 0.59 s, and the AudioX pipeline edges it on CLAP text-audio alignment, 0.309 to 0.302.

In the authors’ blind user study with sound on, people preferred Helix World over EchoWM 67.9% to 20.8%, and over re-dubbed versions of its own video by margins from 67.9%–24.5% to 80.8%–9.6%. With the sound muted the picture looks less special: it tied LingBot-World v2 at 39.5% each and lost to EchoWM 26.3% to 44.7%. The post says plainly that the advantage comes mainly from the audio.

How was Helix World built and trained?

Most video corpora are useless for this task, the team says, because the audio is background music, voiceover or fake stereo with identical channels. They started from 4,100 hours of raw footage (43.9% real-world video, 34.1% game recordings, 22.0% open-source datasets) and filtered it in four stages: heuristics that drop dual-mono audio, cuts and static scenes; standardised 121-frame clips at 24 fps and 768×512 with 48 kHz stereo; a multimodal model that watches and listens to reject music, narration and talking heads; and camera-pose recovery plus three caption tracks. What survived was 3,000 hours in 2.1 million clips, a 73.1% yield.

The bidirectional teacher model builds on the open LTX-2.3 audio-video model. Six-degree-of-freedom camera trajectories enter the visual attention through PRoPE, keyboard actions enter through AdaLN-Zero on the diffusion timestep, and cross-modal attention binds the visual geometry to the audio features. That last step is how camera motion becomes stereo panning. Training the control modules first, then video, then the joint audio-visual backbone lowered camera rotation error from 0.1237 to 0.1144 and translation error from 0.0957 to 0.0823, compared with fine-tuning everything at once.

The teacher sees a whole clip, so it cannot react to live input. The team distilled it into a few-step causal student that generates block by block with a sliding key-value cache and fixed sink frames. Self-forcing with distribution matching alone still drifted, with textures breaking down after 20 to 30 seconds, so they added an online trajectory-distillation loss against the frozen teacher plus a long-horizon tuning stage. Together those raised the student’s WBench score from 77.9 to 79.1 and kept 30-second rollouts coherent.

What can you download, and under which licences?

The NoizAI/HelixWorld repository is Apache-2.0 and contains a standalone inference package for Preview v1. The preview checkpoint on Hugging Face is also tagged Apache-2.0. But the repository’s NOTICE file says the bundled LTX-2.3 code is based on Lightricks’ LTX-2 and that “LTX-2 code and model assets are governed by the LTX-2 Community License Agreement”, and that the downloaded Gemma 3 12B text encoder keeps its own terms. Teams planning commercial use should read all three licences rather than relying on the Apache-2.0 badge.

Hardware is the other gate. The README targets Linux, Python 3.11 and CUDA 12.x, recommends an 80 GB NVIDIA GPU, and says a reviewed BF16 single-GPU run uses about 70 GB of VRAM. The real-time numbers are all from an H800; the paper does not report consumer GPUs. A single run.sh command takes a first frame, a prompt file and an action string such as W for five steps then a right turn, and writes an MP4.

If you only want to see it, the browser demo is the easier route. For context on how quickly open video models are adding sound, compare Kandinsky 6.0 Video’s lip-synced audio, which generates clips rather than a world you steer.

What are the limits, and what should builders watch?

The authors list the limits themselves. Output is 768×512. The public checkpoint is a preview. On picture quality alone Helix World trails the best silent models (79.9 against 82.0), viewers preferred EchoWM with the sound off, and its DeSync score shows onset timing still has room to improve.

Every number here is author-reported, and HelixBench is the authors’ own benchmark, released the same day as the launch post. We read the post, README, NOTICE, arXiv metadata and Hugging Face cards on 12 October 2026; we did not run the model, the demo or the benchmark. An independent HelixBench run on another audio-visual world model would be the useful next data point.

For builders, the interesting idea is that sound can be part of the world state rather than a soundtrack added afterwards. That matters for games, VR and simulation, and potentially for embodied agents that need to hear what is off screen, a gap that latent-world-model work such as AMI Labs’ H-JEPA does not address. Watch for consumer-GPU numbers, a non-preview checkpoint and clearer licence terms for derivatives of the LTX-2.3 base.