Announced 20 Oct 2026 · Sources checked
What was released
The llm-d project has released version 1.0 of its open-source software for running large language models across a cluster of GPUs. The v1.0.0 release went up on GitHub in the early hours of 20 October 2026, and IBM, a co-founder of the project, announced it in San Jose ahead of the PyTorch Conference North America there.
The headline of 1.0 is not speed. It is operations: what IBM calls the “Day 2” work of keeping an inference service running once it is in production. The release adds priority and fairness controls so different customers can share the same GPUs, upgrades that do not drop traffic, a stable v1 configuration API, and routing designed for AI agents that make many calls in a row.
llm-d was started in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, and is now a sandbox project of the Cloud Native Computing Foundation (CNCF), the body that also hosts Kubernetes. IBM says more than 500 contributors from over 25 organisations work on it. The code is under the Apache 2.0 licence.
What llm-d does
llm-d does not replace the inference engine that actually runs a model. It sits around engines such as vLLM and SGLang and decides where each request should go, how replicas are scaled and upgraded, and where cached data lives. It is built for Kubernetes, so teams that already run services there can manage model serving the same way.
The core is a router, which the project calls the endpoint picker. It looks at the state of every model replica and sends each request to the one most likely to answer fastest, often the one that already holds the relevant part of the prompt in its key-value (KV) cache, the memory a model keeps of text it has already processed. Managing that cache well is a business in itself; startups such as Tensormesh now sell enterprise KV cache platforms built on the same idea.
Before 1.0, llm-d already offered prefix-cache and load-aware routing, KV cache offloading to CPU memory and storage, prefill/decode disaggregation (running the prompt-reading and answer-writing phases on separate machines), wide expert parallelism for mixture-of-experts models, and batch and asynchronous processing. Version 1.0 packages those into what IBM's release calls one suite of co-designed components.
Upgrades without downtime
The second theme is zero-downtime operations. The router now runs in a high-availability setup with a hot standby, so a single router failure does not drop traffic, and it supports rolling updates.
The bigger change is for disaggregated serving. A new DisaggregatedSet resource manages the prefill and decode replicas of one model as a single unit, with coordinated rollouts, lockstep upgrades and readiness checks, so an upgrade cannot leave the two halves running mismatched versions. The release notes say this makes disaggregated serving practical in production. It depends on LeaderWorkerSet v0.11.1 or newer, and the project warns that almost all of its guides will break on older versions.
To end confusion over version numbers, every component in the llm-d organisation now carries the same v1.0.0 tag, and the project says it will keep major and minor versions aligned from here on. It is also deprecating its own inference container images in favour of the upstream vLLM, SGLang and TensorRT-LLM images, starting with the ROCm build for AMD GPUs.
Serving agents that come back
The fourth theme is agent-aware serving. An agent working on a task calls the model, pauses to run a tool, then calls again with nearly the same context. If that next call lands on a different replica, the context has to be processed again from scratch.
Version 1.0 adds session-aware routing that keeps a multi-turn conversation close to its cached context, turn-priority scheduling, and support for previous_response_id in the OpenAI-style Responses API, so state can carry across turns. IBM said that in a recent agentic coding test the router captured 97 percent of the cache reuse that was available, according to SiliconANGLE.
Inference engines are being tuned for agent traffic from the other side too: the vLLM project recently sped up Kimi K3 decoding on NVIDIA GB300 systems, and llm-d 1.0 ships with vLLM v0.30.0 to v0.31.0 images, depending on the accelerator, and SGLang 0.5.22.
The cost claims
IBM's release puts numbers on the savings, and they are the project's own. In one use case, IBM Research used open models and llm-d to explore quantum error-correction hardware at 11.5 times lower inference cost than an all-proprietary setup, with a prefix-cache hit rate of about 98 percent and a blended cost of about $0.11 per million tokens. An emailed quote from IBM's Ruchir Puri to SiliconANGLE gave the figure as 11.8 times; the release text says 11.5.
In a second benchmark demonstration, researchers served GLM 5.2, an open-weight model of about 753 billion parameters, on NVIDIA GPUs. IBM says the deployment reached more than 6.6 million output tokens per minute at peak and served up to 3,000 concurrent coding agents with zero preemptions, at an estimated five to ten times lower cost per token than equivalent commercial API pricing, using current cloud rental rates.
Those comparisons depend on assumptions the release does not spell out in full, including which API prices were used and how GPU rental was costed. They show what a well-tuned self-hosted stack can do on traffic that reuses a lot of input, not what a typical team will see on day one.
Who uses it
The release came with quotes from a long list of users. JetBrains said llm-d runs in production behind its new Junie Lite plan, which offers its AI coding agent at no cost, and serves a significant share of that traffic. DigitalOcean said it relies on llm-d for its Serverless Inference and Managed Agents products, serving trillions of tokens. Roblox said it uses llm-d for disaggregated serving and cache-aware routing across cloud and on-premises GPUs, and Cohere said it uses it for both reinforcement learning and production.
The hardware vendors lined up as well. Google Cloud said it is integrating llm-d 1.0 into Google Kubernetes Engine across its TPU and NVIDIA GPU portfolio. NVIDIA said it is bringing llm-d to its Vera Rubin platform, AMD backed it for Instinct GPUs, and CoreWeave said customers can run llm-d on CoreWeave Kubernetes Service alongside NVIDIA Dynamo, NVIDIA's own inference framework.
The timing fits a conference where portability is the theme: at the same PyTorch event, Amazon opened TorchNeuron so PyTorch code runs on Trainium. Red Hat's Brian Stevens framed llm-d the same way, as a standard, hardened core for distributed inference on an open foundation.
How to try it and what to check
llm-d 1.0 is free to download from the project's GitHub organisation, with Helm charts, container images and deployment guides. It assumes a Kubernetes cluster with accelerators and some familiarity with the Gateway API Inference Extension that llm-d's router builds on. The release notes link a testing matrix showing which hardware and guide combinations were validated for 1.0.
Teams moving from earlier versions should read the upgrade notes closely: the router configuration API is now v1, flow control settings have moved, and the disaggregated-serving guide requires a newer LeaderWorkerSet. The project also notes that not everything on its 1.0 goal list was finished.
The open questions are the usual ones for infrastructure software reaching 1.0. The cost and throughput figures come from IBM and its partners rather than independent tests, and the CNCF still lists llm-d as a sandbox project, its earliest maturity stage. A stable API is a promise about the future, and the test of that promise is whether 1.x upgrades stay as smooth as the release says.
Common questions
What is llm-d?
An open-source, Kubernetes-native layer for running large language models across many GPUs. It routes requests, manages caches and scales replicas around inference engines such as vLLM and SGLang.
What is new in llm-d 1.0?
Priority and fairness-aware flow control for shared clusters, router high availability, coordinated prefill/decode upgrades through DisaggregatedSet, a stable v1 configuration API and session-aware routing for agents.
Who is behind llm-d?
It was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA and is now a CNCF sandbox project with more than 500 contributors, according to IBM.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. The cover places the companies’ own logos, taken from public-domain or openly licensed files credited in the caption, on a plain background. It is not a product photograph, a screenshot, or an endorsement.
Our editorial standards
