vLLM speeds up Kimi K3 up to 1.9x on NVIDIA GB300 with agent-assisted kernels

What vLLM shipped for Kimi K3

vLLM, the open-source inference engine that many companies use to serve large language models, merged a new decode path for Moonshot AI's Kimi K3 on 16 October 2026. The code was written at NVIDIA with help from AI coding agents, and NVIDIA engineer Benjamin Chislett says it makes the 2.8-trillion-parameter model generate text 1.5 to 2 times faster on NVIDIA's GB300 GPUs. The pull request adds roughly 12,800 lines of code and was merged about ten hours after it was opened.

The release came with a vLLM blog post titled "Moving Fast and Responsibly" that is as much about process as speed. Chislett writes that "agent-optimized inference code is outpacing our review and maintenance workflow", and the post lays out how the project wants to accept large, machine-assisted performance patches without shipping software that breaks in production.

The same day, vLLM published a second Kimi K3 post from AMD and its partners describing more than 50 merged changes that speed up the model on AMD's Instinct MI355X accelerators. Together the two posts show both GPU makers racing to make one large open-weight model run well on their hardware.

How the specialised decode path works

Kimi K3 is an unusual model to serve. According to vLLM's launch-day post from July, it activates 16 of 896 experts per token, supports up to a million tokens of context and mixes two kinds of attention: Kimi Delta Attention layers that carry a fixed-size recurrent state, and multi-head latent attention layers that keep a compressed history. Generating one token for one user means stepping through more than 90 attention layers and more than 90 mixture-of-experts layers, each of which used to fire many small GPU kernels.

NVIDIA's new path replaces almost all of those operations with custom kernels written in CuTe DSL, NVIDIA's Python-based language for GPU kernels, tuned for decode batches of 1 to 64 across 8 or 16 GB300 GPUs that share one NVLink domain. The blog post shows the effect on a single layer: stock vLLM spends about 130 microseconds and launches 23 kernels, while the specialised path takes about 69 microseconds with 13 kernels.

The post describes four main techniques. Kernels are chosen by batch size, with small-matrix kernels at batch one or two and split-K kernels above that. Computation and GPU-to-GPU communication overlap: the attention output projection is fused with its all-reduce and the residual step that follows, with GPUs writing partial results straight into buffers their neighbours can read. Mixture-of-experts routing work that every GPU used to repeat is split across them and combined. And while GPUs wait on communication, a side kernel prefetches the next layer's weights into cache, since single-user decoding is mostly limited by how fast weights can be read from memory.

The speedups NVIDIA reports

Without speculative decoding, NVIDIA reports that 16 GPUs serving one request produce 256 tokens per second, 1.88 times stock vLLM, and that the gain stays between 1.62 and 1.92 times up to 64 concurrent requests. On eight GPUs the improvement is smaller, from 1.58 times at batch one to 1.31 times at batch 64. At 64 requests on 16 GPUs the whole group produces about 3,606 tokens per second.

With DSpark, the block-diffusion speculative decoding method vLLM already supports for Kimi K3, the headline figure is 842 tokens per second for a single request on 16 GB300 GPUs, against 488 for stock vLLM with the same setup, a 1.73 times speedup. That number uses a synthetic acceptance length of six, meaning the test assumes six drafted tokens are accepted on each step; real prompts usually accept fewer. The pull request's own table lists 838 versus 489 tokens per second for the same configuration, and a smaller 1.36 to 1.75 times gain on 32,000-token inputs with a more realistic acceptance length of 3.78.

On AgentX, a SemiAnalysis benchmark that replays the traffic patterns of real coding-agent sessions with synthetic text, the optimised build reached 403 tokens per second per user at the 90th percentile, versus 224 for an earlier vLLM result. The blog notes that this compares 16 GPUs with the earlier 8-GPU run, so part of the gap comes from extra hardware.

For accuracy, the pull request reports results on AIME25 maths, GPQA-Diamond science questions, GSM8k and the MRCR long-context retrieval test, and says the AIME25, GPQA-Diamond and MRCR scores match stock vLLM within about one standard error, with and without speculative decoding.

Why vLLM wants new review rules for agent-written kernels

The process half of the story matters beyond one model. In an RFC opened on 1 October, Chislett argued that "frontier coding models have become extremely proficient at inference model optimization and efficient GPU kernel development" and that the flow of such pull requests is outpacing reviewers. Chislett proposed a fast track for changes that stay inside one model's code, with automated checks standing in for some line-by-line review. It is the same pressure the Claude Code team described when it doubled its merged pull requests and leaned on AI review to keep reverts down.

The Kimi K3 change is the first built to those rules. All of it lives under the model's own folder, it is switched on only by an explicit flag, and it falls back to the standard code for prefill, mixed batches and large batches. The blog lists four principles: isolate changes, bound the scope without dropping production features such as full-context serving and structured output, validate correctness with tests that actually exercise the new code, and measure performance across batch sizes and context lengths rather than one headline number.

The post also gives two examples of bugs that simple checks missed. An audit found that the compiler had moved activation reads ahead of a synchronisation wait, which was fixed by restricting that load type to static weights. Long-context and long-generation tests surfaced numerical problems in attention and communication kernels that did not show up in standalone kernel tests. A companion Bring-Your-Own-GPU RFC from Red Hat's Lucas Wilkinson proposes a standard evaluation image that contributors run on their own hardware, noting that "AI tools have made producing a PR nearly free, but reviewing one costs as much as or more than it ever did." That cost has a security side too: vLLM published 18 security advisories earlier this month.

AMD's parallel push on MI355X

AMD's post makes a bolder claim. Citing a SemiAnalysis InferenceX snapshot from 9 October, it says an eight-GPU MI355X system serving Kimi K3 with vLLM delivers about 7,567 total tokens per second per GPU at 50 tokens per second per user, 24.4% more than NVIDIA's HGX B300 at 6,083. Using SemiAnalysis's modelled ownership cost of $1.50 versus $2.26 per GPU-hour, AMD says that works out to 87.4% more tokens per dollar.

Those figures compare complete serving stacks on a different NVIDIA system, the B300, and at a throughput-oriented operating point, so they do not directly contradict NVIDIA's GB300 latency results. AMD's changes target the same bottlenecks from another angle: splitting attention history across GPUs with decode context parallelism, removing duplicate mixture-of-experts work, fusing Kimi Delta Attention steps into one kernel, and padding query heads so a faster assembly kernel can be used, which cut time per output token on 32,000-token inputs from 43.8 to 23.6 milliseconds in AMD's test.

Who can use it, and how

The NVIDIA path is opt-in. Operators running Kimi K3 on GB300 with 8 or 16 GPUs enable it with vLLM's additional-config option by setting decode_backend to latency_specialized; it supports DSpark verification with three to seven draft tokens and decode context parallelism for long prompts. No weights change, and requests the path cannot handle go through stock vLLM.

That makes it most relevant to inference providers selling fast, low-concurrency access to frontier-scale open models, such as single-user coding agents, rather than to anyone running Kimi K3 locally. The same design is being applied to other models. A second pull request brings a specialised GB300 decoder to DeepSeek-V4.1-Flash, whose 1017 update shipped on 17 October, and the blog says early results for Meta's Muse Spark Open already exceed 950 tokens per second per user under the same synthetic acceptance setting. A similar low-concurrency path for GLM 5.3 is promised in a follow-up.

Limits and open questions

Every speed number here comes from the vendors that wrote the code: NVIDIA for GB300 and AMD for MI355X. AI Lookout did not run either path, and GB300 systems are rare outside large cloud providers. The biggest headline figure depends on a synthetic acceptance length that real workloads may not reach, and the AgentX comparison mixes 16-GPU and 8-GPU runs.

The pull request's checklist shows that the author left the testing box unticked, noting satisfaction with the validation but uncertainty about how much belongs in vLLM's continuous integration, and the fast-track and Bring-Your-Own-GPU RFCs were both still open when we checked on 18 October. The output-length study in the pull request also found slightly more truncated answers at a 32,000-token cap, which the author calls unclear given high variance. We will update this article if vLLM adopts the fast-track rules, independent benchmarks of the new path appear, or the DeepSeek and GLM versions land.