Short programming tasks have a defined test setting

The benchmark is useful for a defined code-generation setting. It does not directly measure understanding of a large repository, maintainability, or whether a patch meets a changing product requirement.

Candidate count changes the meaning of success

An original model comparison should state how many candidates are generated and how one is selected. A many-attempt result should not be presented as the experience of a reader receiving one answer.

What the metric counts

The model may draw k completions. If any one passes the unit tests, that problem counts as solved for pass@k.

A higher k spends more samples and raises the score even if the first answer is usually wrong. Pass@1, especially with greedy decoding, is the number that matches a user who will read one answer.

Papers that lead with pass@100 are answering a different question: whether the model can hit the tests given many tries.

HumanEval is a few hundred short functions. It does not include a repository, a build, or a vague ticket.

Models have also trained on solutions that leaked. A saturated score is a reason to stop using the set as a headline, not a reason to declare coding solved.

Newer sets exist because of that.

How to say it in a story

Write the score beside k and the sampling method. If a vendor omits k, say it was omitted.

Do not compare their pass@k to someone else’s pass@1. If you need a coding claim about your stack, use your tests.

HumanEval can be a footnote.

What to record before you rely on it

Keep the paper’s definition next to the number you repeat. Do not let a chart axis drop the k.

If you run it yourself, pin the harness version, because scoring scripts differ.

Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.

When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.

Name a person who owns that failure.

Common questions

Is pass@1 the same as accuracy?

It is the fraction of problems whose single sample passed the tests, under the decoding you used. It is not accuracy on writing, reviewing, or modifying a real repository.

Why not always sample many times?

You can, if you have tests and a budget. Users who see one answer do not get that search.

Report the setting they will experience.

Sources & further reading

  1. Evaluating Large Language Models Trained on Code ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories