NVIDIA releases Cosmos3-H-Dreams, a real-time surgical simulator trained on real operations

What NVIDIA released

NVIDIA has released Cosmos3-H-Dreams, an AI model that generates the view through a surgical robot's camera in real time as someone moves the robot's instruments. It was trained on recordings of real gallbladder removals performed with CMR Surgical's Versius robot. NVIDIA announced it on 20 October in a post on the Hugging Face blog, and posted the weights on Hugging Face and the code on GitHub the same day.

The idea is a simulator that does not need a physics engine. Instead of modelling tissue, geometry and contact explicitly, the model has learned from synchronized video and robot movement data how a surgical scene tends to change when the instruments move. Give it one camera frame and a stream of commands for both robot arms, and it produces the video that should follow.

NVIDIA describes it, with the hedge "to our knowledge", as the first real-time generative surgical simulator trained on clinical cholecystectomy recordings that supports live kinematic control. It is a research tool for developing and testing surgical robot software, and NVIDIA states plainly that it is not a validated system for patient care.

How the simulator works

According to the model card, the model takes a single endoscopic frame, up to 16 recent robot commands (1.6 seconds of history) and a stream of future commands, and generates the resulting video one short block at a time. Each step consumes the next four action vectors, 0.4 seconds of motion at 10 updates a second, and produces four new frames. That block-by-block design is what lets it respond while someone is still moving the controls.

Each command has 28 numbers. They cover the position, orientation and jaw of the instrument on each arm, plus electrosurgical energy controls, thumbstick axes and thumbstick buttons. The earlier base model used 20 numbers per step; NVIDIA added the energy and thumbstick inputs for the Versius robot and started them from zero so the pretrained motion knowledge was kept.

In the demos, an operator drives the simulation with a virtual-reality headset and two tracked hand controllers. The GitHub repository serves the generated video to the headset browser and to a PC viewer at the same time, and can also replay recorded movements into an MP4 file. NVIDIA shows two runs that start from the same frame and diverge as the instrument actions differ, with the tissue deforming in response, which is the behaviour a simulator needs to be useful.

The data behind it

Cosmos3-H-Dreams starts from Cosmos3-H-Embodied-Nano, NVIDIA's surgical-robotics version of its Cosmos3-Nano world model. That base model has about 16 billion parameters and pairs a Qwen3-VL-8B reasoning tower with a video diffusion component. It was post-trained on Open-H-Embodiment, a community surgical-robotics dataset hosted by NVIDIA, which its card puts at 750 hours and 120,000 trajectories of video paired with robot kinematics from more than 30 contributing organizations, among them CMR, Johns Hopkins, Stanford and Vanderbilt.

The specialization step used proprietary CMR Surgical clinical data. The source pool was 4,792 recorded episodes, about 17 million frames at 60 frames a second in 1080p. NVIDIA kept only clips in which exactly two instruments were visible and engaged, normally a grasper on the left arm and a monopolar hook on the right, and dropped windows where the camera was being moved, so the model learns instrument-driven changes rather than camera pans.

After filtering, the training set was 64 recordings and roughly 526 unique minutes of video, resampled to 10 frames a second at 832 by 480 pixels. Ten recordings, about 10.6 minutes, were held back for testing. The card describes the material as intra-abdominal video and robot telemetry from procedures on human patients, showing anatomy and instruments but no faces or other identifying views.

From a slow teacher to a fast student

Generating video fast enough to keep up with a human hand is the hard part. NVIDIA first fine-tuned a slow, high-quality teacher model that looks at whole clips in both directions, first on 24-frame windows (2.4 seconds) and then on 72-frame windows (7.2 seconds). The teacher needs 35 sampling steps per output, far too many for live use.

It then distilled that teacher into a causal student that only looks backwards and generates in short blocks as new actions arrive. Training ran in two stages: causal teacher forcing on recorded footage, then Self-Forcing, in which the student learns from its own generated history under the guidance of the teacher and a learned critic. That second stage targets a known weakness of video models, where small errors pile up over a long rollout until instruments change shape or anatomy drifts.

Two checkpoints are released: one that uses four denoising steps per block and a separately trained one that uses two, trading some visual quality for speed. For long sessions, the integration with FlashDreams, NVIDIA's streaming-inference library for interactive video models, uses an adaptation of the FadeMem method to consolidate the model's growing memory cache instead of letting it grow without limit.

How fast it runs

NVIDIA's own measurements put the four-step model at 30.0 frames a second and the two-step model at 43.2 on an RTX PRO 6000 Blackwell Workstation Edition, both at the native 832 by 480 resolution. The lower-power Max-Q edition of the same card reached 20.2 and 31.2 frames a second. On the DGX Spark desktop system, the two-step model ran at 16.9 frames a second, but only at a reduced 512 by 288 resolution.

Those are workstation-class GPUs, not laptops. The repository says the default configuration needs more than 32 GB of video memory, and the card lists NVIDIA Blackwell and Hopper as supported architectures, with H100 also tested. Only bf16 precision has been qualified; NVIDIA has not tested other precisions or quantized versions.

What researchers can do with it

NVIDIA lists three intended uses: evaluating surgical robot control policies in simulation before they touch hardware, interactive teleoperated exploration of the model, and generating synthetic training data for robotic cholecystectomy. It fits a wider push at the company to train and test robot software inside learned simulators, alongside research such as Long-WAM's longer camera memory for robot policies and the ASPIRE framework in which coding agents write robot skills.

The release also includes a step-by-step recipe for fine-tuning a teacher on other data and distilling it into a real-time student, so groups with their own synchronized video and kinematics can try the same approach with other robots or procedures. Controllers have to follow the same action conventions and normalization used in training.

NVIDIA's list of next steps shows how far this is from a finished product: cover the whole procedure rather than selected segments, combine learned visuals with explicit geometry and tissue mechanics, have surgeons judge anatomical accuracy and turn that into benchmarks, support more than two instruments, and eventually connect autonomous surgical policies to the simulator in a closed loop.

Licences and access

The model weights are covered by the NVIDIA Open Model License, which the card says allows commercial and non-commercial use. The simulator code in the GitHub repository is Apache 2.0. The example starting frame bundled with the code, the image at the top of this article, comes from the Open-H-Embodiment dataset under a Creative Commons Attribution 4.0 licence, credited to CMR and the dataset contributors.

The CMR clinical recordings used for fine-tuning are proprietary and were not released, so outsiders cannot retrain the specialist model from scratch on the same footage. Cosmos3-H-Dreams succeeds Cosmos-H-Dreams, an August 2026 model built on Cosmos-Predict2.5 that simulated tabletop suturing with the open-source dVRK research robot rather than clinical scenes.

What it is not

NVIDIA is careful about the limits, and readers should be too. Looking realistic is not the same as being biomechanically accurate, the blog post says, and the realism is "a starting point for validation, not proof". The published evaluation is NVIDIA's own: two fixed held-out panels of 48 visual-quality cases and 12 action-control cases each, plus rotation probes and 30.4-second continuous rollouts reviewed by eye. No independent surgeons or outside groups have yet published an assessment.

The model is tied to one robot, one procedure and two specific instruments, and the speeds above come from NVIDIA's hardware tests. Generative world models are spreading fast in robotics, including Dream Labs' Dream-1 for two-armed robots, and a German government pact with NVIDIA on open physical AI models signed this week. Cosmos3-H-Dreams is a notable step because it moves that idea inside the human body, but for now it is a research simulator, not a training tool approved for surgeons or a system that touches patients.