Match the task, attempts, tools, and budget

An original comparison checklist can record attempts, context, tool access, latency, and whether results were measured by the vendor or another party. If those conditions differ, explain the mismatch before declaring a winner.

Use benchmarks to shortlist, then test your workflow

A model that leads one benchmark may still be unsuitable for your workflow. Use benchmarks to form a shortlist, then evaluate the task and constraints you actually have.

What a careful sentence includes

Name the benchmark. Say whether a higher number is better.

Say who ran it: the company, a third party, or a leaderboard with a published harness. Say the date.

“State of the art” without those words is not a fact you can check next month, because the table will have moved and the phrase will still be on the page.

Watch for a subset. A score on an easy slice, or with tools the baseline was not allowed to use, is a different test.

If the post does not say, do not guess that the conditions matched. Write that the conditions were not stated.

What you owe the reader

One clause on what the test does not measure. A coding benchmark is not a measure of writing.

A multiple-choice test is not a measure of whether the model should send an email. You can report the number and still refuse the leap.

The refusal is the useful part.

What to record before you rely on it

Keep the link to the harness or the paper, not only the blog. If the link is missing, say the company stated the score and did not show the method in the post you read.

Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.

When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.

Name a person who owns that failure.

Common questions

Should we ignore vendor benchmarks?

No. They are information about what the vendor measured. Ignore the leap from that measurement to an unrelated job.

Attribute the number.

What if two posts disagree?

Report both, with dates and methods. Do not average them into a winner.

Disagreement is often a difference in harness, and the harness is the story.

Sources & further reading

  1. Holistic Evaluation of Language Models ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories