Sources checked
Repository issues test the surrounding agent system
A repository issue can require locating relevant files, understanding an existing codebase, and producing a change that passes tests. A model’s text-only ability is only one component of an agent’s result in that environment.
Scores depend on tools and attempt budgets
When comparing scores, check the benchmark variant, allowed attempts, tool access, and evaluation method. Do not compare numbers with different setups as if they measured the same experiment.
What the items are
Jimenez and colleagues built tasks from GitHub issues and the tests that accompanied the fixes. The agent sees the repo and the issue.
Success is a patch that passes the tests they hold. That is harder than completing a function in a single file, and it is still narrower than a job.
The tests are the tests. A patch can pass them and be a bad change.
A patch can fail them and contain a useful idea. The benchmark does not score the idea.
Later variants, including multimodal or verified subsets, are different sets. A number on one is not a number on the other.
Agent scaffolds differ too: the tools, the retries, the model. A leaderboard that hides the scaffold is hiding the system.
Ask for both.
How to use the number
Use it to compare systems that publish the same split and a reproducible scaffold. Do not use it to hire, and do not use it to skip your own repo’s tests.
If you evaluate an agent on your code, build a set like this: real issues, hidden tests, a clean checkout per attempt. Report the scaffold.
The benchmark paper is the template, not the score of your agent.
What to record before you rely on it
Cite the split and the scaffold next to any SWE-bench number. If you did not run it, say whose number it is and the date.
Keep your internal set separate so the public score cannot launder it.
Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.
When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.
Name a person who owns that failure.
Common questions
Does a high score mean the agent understands the codebase?
It means the patches passed the tests in that harness, under that scaffold. Understanding is not the metric.
A later issue in the same repo can still fail.
Why do scores move so fast?
Models change, and scaffolds change. A jump that arrives with a new wrapper is a wrapper result until someone shows the same wrapper on the old model.
Look for that ablation.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





