Modal trains RL with mixed-precision samplers across six GPU types

What did Modal publish?

In a 13 October 2026 Modal blog post, Nan Jiang asked whether large-model reinforcement learning can keep training stable when the samplers that generate rollouts no longer match the trainer’s accelerator or numeric precision. Outside RL, serving stacks already chase FP8, MXFP4, NVFP4 and specialist prefill hardware; inside RL, samplers usually stay glued to the trainer’s fleet so the two policies stay numerically close.

Sampling already consumes most of the GPUs in large RL runs—Modal cites 80–85%—so locking that fleet to trainer-class accelerators leaves money on the table. The post’s claim is that off-policy correction, which labs already use for asynchronous staleness, also covers precision and accelerator mismatch, and that Modal’s autoscaling GPUs make a heterogeneous sampler pool operable mid-run.

This is an engineering research result with public plots, not a product SKU launch. There is no new Modal pricing tier announced in the piece and no third-party reproduction. The dated object is the method plus Modal’s measured curves on one coding RL setup.

How did the mixed-sampler setup actually run?

They train Qwen3.6-35B-A3B on agentic coding with up to 256K context using Xiaomi’s 2.7K MiMo-V2.6 repository tasks after scrubbing history that could leak the reference fix. Each episode lets the model patch an issue over multiple turns inside a Modal Sandbox and scores 1 only if tests pass in a fresh sandbox. Evaluation is SWE-bench Pro V2, including a hard HARD-51 subset, always in BF16.

The stack is Miles for RL, SGLang samplers and a Megatron-LM trainer that updates the BF16 policy on B200 GPUs. Training is fully asynchronous with no staleness cap; each update uses 128 prompts with 8 trajectories. Samplers keep their own accelerators, precisions and kernels, and the trainer does not replay MoE routing decisions.

The mixed fleet spans six accelerator classes: B200 and B300 serve NVFP4 weights; H100 and H200 serve FP8 and BF16; A100 and RTX PRO 6000 serve BF16. Each group runs its own kernels with an FP8 KV cache. A router assigns agent sessions in proportion to serving capacity. The homogeneous control uses only B200 BF16 weights and a BF16 KV cache so only kernels and staleness separate samplers from the trainer.

That heterogeneous serving picture is closer to how production inference already looks than a single-sku RL cluster. For adjacent infrastructure that attacks repeated prompt cost on the serving side, see AI Lookout’s coverage of Tensormesh’s enterprise KV-cache platform.

What stabilized training when mismatch jumped?

Without correction, GRPO collapses on both setups: quickly on mixed samplers, later on homogeneous ones. Collapse signs are familiar—training reward falls, sampler–trainer KL grows 3× or more, and episodes die on repeated format errors when the model stops calling tools. Matching the trainer’s accelerator only delays the failure.

Modal layers three fixes. Masked importance sampling reweights each token by the training-to-sampling probability ratio and zeroes ratios outside a band. Score centering subtracts the expected log-prob gradient under the sampling policy so constant advantages stop drifting the trainer toward the samplers. Top-p sampling with mask replay keeps rare tokens out of the data and forces both policies onto the same candidate support.

Only SC + MIS, with or without top-p, trained stably to the end in their sweeps. Pass@4 rose over roughly the first 100–140 steps then leveled near 65–75% on HARD-51 in their plots; the full SWE-bench Pro V2 set showed the same pattern from a higher start. Ablations that used only one piece of the correction delayed collapse but finished at or below the base model.

Average sampler–trainer KL on the mixed run was about 4.7× the homogeneous run even while quality stayed aligned under SC + MIS + top-p. Modal’s reading is that stability depends more on keeping mismatch bounded than on keeping it tiny—an important shift if you want serving-class optimizations inside RL.

How did mixed samplers compare on cost and SWE-bench Pro V2?

With the full correction, mixed and homogeneous pass@4 curves stay close throughout training on SWE-bench Pro V2. Training reward and gradient norms follow similar paths despite the larger KL. Entropy matches until about step 160, then rises on the homogeneous run and stays flatter on mixed.

Cost is where Modal leans in. For most of training, sampler cost per update was about 15% lower on the mixed fleet than on homogeneous B200 BF16 sampling. They stress the experiment was not cost-minimized: each sampler group can be tuned alone, and the mix can shift toward cheaper accelerators once stability is not the open question.

Idle-capacity angles elsewhere in the industry rhyme with the same theme of not leaving accelerators stranded. AI Lookout’s piece on Apple’s Minions GPU cloud on idle Apple Silicon is a different stack, but it is another reminder that utilization policy is becoming part of the model story, not just a cluster footnote.

Why this matters for RL infrastructure

If correction really absorbs precision and accelerator mismatch, RL sampling no longer needs a homogeneous mirror of the trainer. Fleets can burst on whatever GPUs can serve the policy, grow mid-run as trajectories lengthen, and adopt speculative decoding or prefill/decode splits without forcing the trainer to match every trick.

Trainers can likewise move to their own best accelerators and low-precision recipes without dragging samplers along. As vendors multiply number formats across generations, that decoupling is less a nice-to-have and more a requirement for people who do not own a single-sku superpod.

The post is also careful about what correction cannot fix. Multi-turn detokenize/retokenize bugs and top-p support mismatches still need system matching; Modal keeps token-in-token-out and mask replay for those. MoE routing replay and indexer replay remain costly options they did not take, accepting a biased target rather than paying that overhead.

What to verify before copying the recipe

Treat SWE-bench Pro V2 pass@4, HARD-51 curves, KL multiples and the 15% cost delta as Modal’s measurements on Qwen3.6-35B-A3B with Miles/SGLang/Megatron. Different models, rewards or sandbox harnesses can move every plot.

Confirm your own off-policy stack implements MIS masking bands, score centering and—if you sample with top-p—mask replay before declaring mixed fleets “solved.” The ablations show partial fixes fail.

Do not read the result as permission to ignore bitwise training–inference research. Modal argues you often should not pay for full bitwise matching; that is a cost/quality trade, not a claim that matching never helps.

Watch for follow-ups that put the trainer in FP8/NVFP4, shift the sampler mix toward cheapest stable groups, or test non-coding rewards. Those would be update material for this story rather than brand-new launches.