Sources checked
What does a coding benchmark actually measure?
In Evaluating Large Language Models Trained on Code, Mark Chen and colleagues define HumanEval as 164 handwritten programming problems. Each one has a function signature, a docstring, a body, and unit tests, with an average of 7.7 tests. They wrote the problems by hand because models trained on GitHub could otherwise be scored on solutions that were already public. They also report that programs which fail the tests can still have a higher BLEU score than programs which pass, which is why they use tests rather than text overlap.
On that set, their Codex model solved 28.8 percent of the problems. GPT-3 solved none, and GPT-J solved 11.4 percent. With 100 samples per problem they solved 70.2 percent. Repeated sampling is the idea behind pass@k: the chance that at least one of k samples passes. HumanEval and pass@k walks through that score. A 2021 number is not a 2026 ranking.
What should you compare instead of a single winner?
Start with the release you are allowed to run. Open weights vs open source is the difference between a weight file and the Open Source Initiative’s definition. Then test tasks from your repository, including ones the model should refuse or leave unfinished. A vendor’s coding claim is evidence about the vendor’s evaluation, not about your build.
| Check | Why it changes the choice |
|---|---|
| License and weights | You may not be allowed to run or ship the checkpoint you liked in a demo |
| Tests from your repo | HumanEval problems are not your private APIs or build |
| One sample vs many | pass@k can look strong because the model had several tries |
| Tool loop | Editing a repository is an agent task, not only a completion |
Why this guide does not name a winner
AiLookout has not run a bake-off of current open coding models. Naming one would invent a result. Use the paper’s lesson instead: functional tests on problems the model was unlikely to have memorized, reported with the number of samples, and read next to the license.
Common questions
Is the model with the highest HumanEval score the best open-source coding model?
Not by itself. The score is functional correctness on that handwritten set, for the sample count the authors report. Your repository and your license can point somewhere else.
Did the Codex paper rank today’s open models?
No. It compared Codex with GPT-3 and GPT-J on HumanEval. Later models are outside that paper.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards
