Ant Group and vLLM launch Humming, a kernel library for quantized inference

What Ant Group and vLLM announced

Ant Group and the vLLM community on 21 October 2026 formally announced Humming, a library of GPU kernels for running quantized large language models, in a long post on the vLLM blog. Humming handles the matrix multiplications at the heart of inference when model weights, and sometimes activations, are stored at reduced precision such as 8, 4 or even 1 to 3 bits. It covers both ordinary dense layers and the expert layers of mixture-of-experts (MoE) models, and the code is on GitHub under the Apache 2.0 licence.

The announcement makes three claims. Humming runs on every NVIDIA GPU architecture from Turing (2018, compute capability SM75) onward, and also on CUDA-compatible accelerators such as the Zhenwu 810E from T-Head, Alibaba's chip unit. It supports a wide range of weight and activation formats through one shared design rather than one hand-written kernel per format. And it outperforms the kernels vLLM and others use today, namely vLLM's Marlin, FlashInfer and DeepSeek's DeepGEMM, across many of the workloads the authors tested.

The launch is more of a formal debut than a first release. The package has been on PyPI as humming-kernels since May 2026, version 0.1.18 was tagged on 21 October, and vLLM 0.29.0, released on 9 September, already listed Humming MoE backends among its new quantization options. The blog post is the first full account of how the library works and how fast its authors say it is. Most of the code is by Ant Group engineer Jinzhen Lin, with contributions from vLLM maintainers including Michael Goin.

Why quantized inference needs so many kernels

Quantization stores a model's numbers with fewer bits, which cuts memory use and the amount of data a GPU has to move for every token (our quantization explainer covers the basics). The saving matters most for MoE models: each token only activates a few experts, but all of them have to sit in memory. The Humming post uses DeepSeek-V3 as its example, with 671 billion parameters of which 37 billion are active per token.

The catch is that quantization is not one format. The post lists common setups such as WNA16, W8A8, W4A8 and W4A4, where W and A give the bit widths of weights and activations. Even at the same width, INT4 and NVIDIA's NVFP4 use different number formats and scaling. Group sizes, scale data types, symmetric or asymmetric schemes and zero points multiply the options, and each GPU generation supports a different subset in hardware.

Memory savings only turn into speed if a kernel can unpack and multiply those numbers efficiently. Until now, inference engines have tended to add a dedicated kernel for each combination: Marlin for 4-bit weights with 16-bit activations, DeepGEMM for FP8, separate paths for newer formats. The Humming authors argue that this creates a growing maintenance burden, because every new format needs its own preprocessing, kernel selection, framework adapters and tests.

How Humming builds a kernel on demand

Humming splits the problem in three. First, it converts models exported by different quantization tools into a single standard description that records weight format, scaling granularity, scale type, zero points and secondary scales as separate parameters. Second, it combines that description with the layer shape, the type of matrix multiplication and the target GPU to pick a configuration, including tile shapes, pipeline depth, warp specialisation, memory-access method and which Tensor Core instructions to use. Third, it compiles a kernel just in time from shared CUDA building blocks, using NVIDIA's NVCC or NVRTC compilers, and caches it.

The building blocks are grouped by what varies. Numeric components deal with formats and bit widths, memory and compute components with GPU architectures, and scheduling components with dense versus MoE work. Adding a format or a chip means extending one group rather than writing a new kernel. On Hopper GPUs, which lack native FP4 Tensor Core support, Humming dequantizes MXFP4 weights to FP8 or INT8 inside the kernel; on Blackwell it uses the native microscaling formats.

Compilation speed is part of the pitch. In the authors' test on an H200 server with CUDA 13.4 and caches off, compiling one W4A16 kernel took 23.35 seconds with NVIDIA's CUTLASS and NVCC, against 1.82 seconds for Humming with NVCC or 1.88 seconds with NVRTC. Compiling every kernel needed for one layer across all batch sizes takes about two to four seconds during model warm-up, they say. The post also notes that Humming exposes two Blackwell Tensor Core formats, float4e0m3 and float8e3m4, that community researchers found are supported in hardware but not exposed in NVIDIA's PTX type system; Rubin GPUs do not have them.

What the benchmarks show

The kernel microbenchmarks in the post were run by the vLLM community across batch sizes from 1 to 8,192, with warmed-up kernels and a cold L2 cache. For dense W4A16, the most common setup, the authors report roughly three to four times the speed of cuBLAS at small batch sizes on several GPUs, about twice Marlin's performance on large H200 batches and more than three times Marlin in some B300 cases. For MoE layers with 4-bit weights, they report about twice Marlin on H20 and H200 at medium to large batches and up to about five times on B300 with the grouped path.

Against DeepSeek's DeepGEMM, Humming's FP8 W8A8 path is reported at about 2.6 times on H20 and 2.5 times on H200 at small and medium batches, and its W4A8 path with MXFP4 weights at about 4.2 and 4.5 times. On the B300 the gains over DeepGEMM shrink to about 1.25 to 1.6 times, and on the SM120 and SM121 chips used in workstation and personal-computing products, the lead over FlashInfer is about 19 percent for W4A8 and 9 percent for W4A4 at small batches.

The figure that matters most to operators is an end-to-end serving test. The authors ran DeepSeek-V4.1-Flash on eight NVIDIA H20 GPUs with tensor parallelism of eight, kept the model's MXFP4 weights unchanged, and switched only the MoE layers from Marlin W4A16 to Humming W4A8 with FP8 activations. Throughput rose by 14 percent with one request in flight and by 43 percent at 64 concurrent requests, where output went from 442 to 634 tokens per second. Mean time to first token fell by 25 to 32 percent across the range.

How to use it in vLLM

For vLLM users there is little to install. Since vLLM 0.29.0, CUDA builds pull in Humming as a dependency, and the --linear-backend humming and --moe-backend humming flags select it for dense and MoE layers. A separate --quantization humming option loads formats that vLLM itself no longer supports or does not yet support, such as 3-bit GPTQ checkpoints. If a GPU cannot compute in a checkpoint's activation format, Humming falls back to a higher-precision type, so the same quantized file can run across GPU generations.

Outside vLLM, Humming installs with pip install humming-kernels, with optional extras that bundle the CUDA 12 or CUDA 13 compiler. It needs PyTorch and a CUDA compiler and nothing else; PyPI lists the 0.1.18 wheels at about 510 KB. A short Python example in the README quantizes an 8,192 by 8,192 layer to 6-bit integers and runs it through a HummingLayer object.

What it means for teams serving models

Kernels are where much of the recent speed in open inference has come from. In the past week DeepSeek open-sourced DeepInfer, the engine behind its V4.1-Flash service, and vLLM published kernels that speed up Kimi K3 on GB300. Humming is less flashy than either, but it targets a broader problem: the long tail of quantization formats and older GPUs that teams actually have in their racks.

Support back to Turing means T4 and RTX 20-series cards can use the same code path as B300s. The H20 results are telling too. The H20 is the cut-down Hopper chip NVIDIA designed for the Chinese market, and support for T-Head's Zhenwu accelerators points the same way: Chinese operators need to get more out of the hardware they can actually buy. For anyone running mixture-of-experts models in 4-bit or FP8, a drop-in backend that the authors say lifts throughput by a third or more at typical concurrency is worth testing.

Limitations and open questions

Every number in the announcement comes from the Humming authors and the vLLM community, which also maintains the Marlin kernels used as a baseline. AI Lookout has not run the benchmarks. The end-to-end test covers one model on one GPU type, and it compares Humming's W4A8 path with FP8 activations against Marlin's W4A16 path with BF16 activations, so part of the gain comes from the different activation precision, not only from a faster kernel. The post does not report accuracy differences between those settings.

Results on other hardware are mixed by design. The gains on Blackwell are smaller than on Hopper, and the authors say heuristics cannot always find the best configuration; an autotuner is on their roadmap, along with Rubin features, T-Head's Zhenwu M890 and quantized all-to-all communication for expert parallelism. The announcement lists no AMD or Intel support. The library is at version 0.1, and the README and the blog differ on package size, which suggests documentation is still catching up with the code.