Sources checked
Draft acceptance determines the benefit
The benefit depends partly on how often the target accepts the draft and how expensive the draft is to generate. If the draft frequently disagrees, coordination can reduce the expected speed benefit.
Compare the same target and sampling settings
An original serving test should compare the same target model and sampling settings with and without speculation. Check throughput, latency, and whether the implementation preserves the intended behavior.
How the check works
The draft model writes a short run of tokens. The large model scores them in one pass and accepts a prefix, then samples a correction where it rejects.
Papers from Leviathan and colleagues and from Chen and colleagues showed you can do this without changing the large model’s output distribution, in exact arithmetic. You get the large model’s samples, faster, when the draft is often right.
When the draft is often wrong, you waste the draft and may not win.
The exactness depends on the implementation. A production server that “speculates” with a heuristic, a quantized draft, or an early exit may not be the algorithm in the papers.
If a vendor claims speculative decoding, ask whether the output distribution matches the target model. If they will not say, do not promise users the answers are identical to the slow path.
What to measure
Measure latency at your batch size and sequence length, and measure whether outputs actually match the non-speculative setting on a fixed seed, if you need them to. A speedup that changes answers is a different model.
Report it as a different model. Cost can fall because the large model runs fewer steps, or it can rise because you now run two models.
Count both.
What to record before you rely on it
Record the draft model, the target model, and whether you verified distribution match or only latency. Cite the papers for the method, not for a vendor’s speedup.
Keep a paired output sample. Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.
When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.
Name a person who owns that failure.
Common questions
Is the draft model the one answering?
Only for the tokens the target accepts. A correct implementation lets the target overwrite the draft.
If you cannot confirm that, you do not know which model spoke.
Should we speculate on every endpoint?
Only where the draft agrees often enough to beat the simple path, and where a mismatch would be noticed. Measure it.
A blanket flag is how a careful path becomes an unreviewed one.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





