Sources checked
Aggregation needs comparable final answers
The approach depends on enough diversity between samples and an answer that can be compared or aggregated. Open-ended research questions are harder to reduce to a simple vote than a calculation with a clear final value.
What the method does
Instead of taking one greedy decode, you sample multiple completions and choose the final answer that appears most often. On the math and commonsense sets they tested, the vote beat a single greedy path.
The cost is several generations. The assumption is that the true answer is a attractor and the mistakes scatter.
When the mistake is shared, the vote becomes confident and wrong.
That failure mode is the whole caution. A model that always misreads a table will misread it in every sample.
Self-consistency does not add a table. It adds a majority.
If you cannot afford to check the majority, you cannot afford to present it as verified.
When it is worth the tokens
It is worth it on closed answers where you can see the vote split: a number, a label, a short span. It is a poor fit for long prose, where “the same answer” is undefined.
Cap the samples. Log the distribution, not only the winner.
A 3-to-2 vote is not the same object as a 10-to-0 vote, and the paper’s gains do not make them the same.
What to record before you rely on it
Record the sample count, the temperature, and the vote split. Cite Wang and colleagues if you cite the method.
Do not describe a unanimous wrong answer as consistent reasoning. Call it a consistent error.
Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.
When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.
Name a person who owns that failure.
Common questions
Is this the same as chain-of-thought?
Chain-of-thought is one path that shows steps. Self-consistency is several paths and a vote.
You can do the first without the second. The second costs more.
Can the model vote on open-ended text?
Only if you define equality. String match will treat paraphrases as disagreements.
A second model judging equality is another system, with its own errors. Prefer votes on short answers.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





