DeepSeek open-sources DeepInfer, the engine behind its V4.1 Flash service

What DeepSeek released

DeepSeek has open-sourced DeepInfer, the inference framework that it says runs its own large-scale production services. The code went up in the deepseek-ai/DeepInfer repository on GitHub on 20 October 2026, with an initial commit in the early morning European time and documentation updates through the day, under the Apache 2.0 licence.

This first release supports one model: DeepSeek-V4.1-Flash, whose 1017 checkpoint shipped as open weights last week. The README's example downloads that checkpoint from Hugging Face on first run. DeepSeek says inference code for future models, and later performance work, will also be published in DeepInfer.

Alongside the engine, DeepSeek published a long deployment Q&A that walks through how its own V4.1 Flash service is built, and pointed to deepseek-recipe, an MIT-licensed component that converts between common chat API formats and the model's token sequences. DeepSeek says recipe's logic matches the official DeepSeek API, so third-party hosts can reproduce the same API behaviour.

What is inside DeepInfer

DeepInfer is an engine, the software that actually executes a model on accelerators, batches requests and samples tokens. The README describes this release as offline inference with the core features needed for serving at scale. Those features include optimised kernels, attention data parallelism, mixture-of-experts expert parallelism with expert load balancing (EPLB), asynchronous scheduling, structured outputs and speculative decoding.

The kernels are DeepSeek's own previously open-sourced libraries, FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect, plus custom kernels written in TileLang. DeepSeek says the internal versions differ little from the public ones; one hand-tuned optimisation missing from public FlashMLA makes its decoding kernel 5 to 10 percent slower, and end-to-end performance is within 5 percent.

Two features target reproducibility. Batch-invariant inference keeps a request's results identical no matter which other requests share its batch, and deterministic sampling gives identical outputs for identical inputs and a fixed seed. Both are NVIDIA-only for now. Structured outputs can be constrained to a regular expression, a JSON schema or an EBNF grammar.

The README credits vLLM as the original base of DeepInfer, with inspiration from SGLang and NVIDIA's TensorRT-LLM.

NVIDIA SM100 and Huawei Ascend 950

DeepInfer supports two hardware families: NVIDIA SM100 GPUs, the Blackwell data-centre generation, and Huawei Ascend 950 NPUs. Building it needs Linux, Python 3.12 and GCC 13 or newer, plus CUDA 13 for NVIDIA or Huawei's CANN 9.2 for Ascend. An environment variable selects the platform at build time.

The deployment notes say one of the system's starting goals was to run on accelerators from multiple vendors and make full use of each. Shipping Ascend support in the same release as NVIDIA support gives developers in China, who face limits on buying NVIDIA's top chips, an official path to DeepSeek-tuned serving on domestic hardware.

The release lands during a political fight over the company. A day earlier, the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products, including tools built on its models. Open-source code on GitHub is a different thing from DeepSeek's hosted apps, but agencies will read the order's scope carefully before touching either.

How DeepSeek's own service is built

The deployment Q&A is the most detailed part of the release. DeepSeek's service splits work into three stages on separate machines: a vision encoder for images, prefill for reading the prompt, and decode for writing the answer, plus dedicated storage nodes. Load balancers sit in front of each stage, and an entrypoint handles requests after recipe turns them into tokens.

Prefill runs on ordinary eight-GPU machines with small expert-parallel groups, two to eight ways. Decode uses wide expert parallelism across 64 to 128 GPUs in one scale-up domain, which DeepSeek says cuts per-step memory traffic in the expert layers and raises throughput. Tensor parallelism is not used for now.

Because decode is recommended at 64 GPUs or more, DeepSeek says the smallest efficient unit is 64 GPUs of prefill plus 64 of decode. That is a hyperscale configuration. It also says that because V4.1 Flash compresses its key-value (KV) cache heavily, prefill and decode can sit in different data centres, budgeting about 400 Gbps of links per eight GPUs, and that its own nodes run in adjacent data centres.

DeepSeek says its load balancing reaches roughly 80 percent of a perfectly balanced offline benchmark.

A cache that lasts for days

Another starting goal was to keep KV cache, the model's stored memory of text it has already processed, for more than 72 hours, so repeated context does not have to be recomputed. DeepSeek stores it on 3FS, its own distributed file system, with about 100 PiB of SSD capacity and 10+2 erasure coding.

Cached data lives for three days for agent traffic and seven days for chatbot traffic. DeepSeek explains that chatbot traffic has a heavier long tail, where a small amount of cache is read many times, and that a three-day limit would cost chatbot traffic 0.5 to 1 percent of its cache hit rate.

The Q&A also recommends storage servers with 20 SSDs of 30.73 TB each and about 14.4 TB of logical cache per online GPU. Startups such as Tensormesh sell enterprise KV cache management because this layer is hard to build. DeepSeek's cache store and load balancers are tied to its internal infrastructure and are not fully open-sourced; the Q&A and a Zhihu blog post describe their design instead.

Smarter speculative decoding

Speculative decoding has a small draft model propose several tokens that the main model then checks in one pass. DeepSeek's version, DSpark, adds a confidence head that predicts how likely each draft token is to be accepted, and spends verification compute only on the most promising ones.

In production the draft model proposes five tokens at a time. DeepSeek's table shows dynamic verification cuts the tokens checked per step by about 30 percent while the average accepted length barely moves, 3.29 versus 3.20. It puts the gain at 4 to 7 percent throughput at its usual operating point of about 60 concurrent sequences per GPU rank, rising to 11 percent at 96, with a 1.5 percent loss at 16.

A separate trick, bounded replay for the model's sliding-window attention layers, recomputes only the last 128 tokens when that part of the cache is missing. DeepSeek says the decoder version nearly halves total prefill computation.

The production numbers DeepSeek disclosed

DeepSeek attached two business figures to the release. It says a single cluster currently serves 155 trillion tokens per day, with overall revenue of $38 per GPU-hour at its off-peak valley pricing. Those are the company's own numbers; it did not say how many GPUs the cluster has or how the revenue figure was calculated.

The README's benchmark charts were measured on four SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and a sweep script is included so others can reproduce them. A separate chart compares 64-way and 8-way expert parallelism for decode at a 131,072-token context.

The repository also includes evaluation scripts for AIME 2026, Humanity's Last Exam and GPQA Diamond, built on the offline batch API. DeepSeek did not publish fresh scores for them with the release.

What is missing and who can use it

The README is clear about what is not included. A complete service still needs a KV cache system, full disaggregation of the vision, prefill and decode stages, and load balancing across data-parallel replicas. DeepSeek says most of the engine functionality for disaggregation is already implemented.

That makes DeepInfer most useful today for cloud providers and large labs that already run SM100 or Ascend 950 clusters and want to host DeepSeek models with the company's own optimisations, and for engineers who want to study how a frontier lab serves at scale. Smaller teams will usually be better served by vLLM or SGLang, which support many models and modest hardware.

The release fits a broader push to open up production inference. The same day, the CNCF project llm-d reached version 1.0 with tools for running shared, always-on model serving on Kubernetes. Coverage such as Crypto Briefing's report framed DeepInfer as a way for enterprises to cut compute and energy costs; whether it does that outside DeepSeek's own data centres is something only independent deployments will show.