Sources checked
Repository issues test the surrounding agent system
A repository issue can require locating relevant files, understanding an existing codebase, and producing a change that passes tests. A model’s text-only ability is only one component of an agent’s result in that environment.
Scores depend on tools and attempt budgets
When comparing scores, check the benchmark variant, allowed attempts, tool access, and evaluation method. Do not compare numbers with different setups as if they measured the same experiment.
What to remember
Review failures beyond the test score.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards
