AgentSWE benchmark finds coding agents rarely build the agent you asked for

What is AgentSWE?

AgentSWE, posted to arXiv on 9 October as paper 2610.13361, asks a question most coding benchmarks skip: when you hand a coding agent a written requirement for an agent, does the agent it delivers actually do the job? The authors are Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang and Zheng Liu, from the Beijing Academy of Artificial Intelligence (BAAI), the Institute of Automation at the Chinese Academy of Sciences, and Renmin University of China.

The benchmark has 25 tasks spread over three stages of an agent’s life. Ten Creation tasks start from an empty workspace, with jobs such as desktop GUI automation, formal theorem proving, repository bug repair, scientific PDF translation and web research reports. Ten Editing tasks add capabilities to pinned production agents, including OpenHands, Aider, Codex, Claude Code, DeepTutor and OpenWiki. Five Optimization tasks push a working agent on BrowseComp, τ³ retail, Terminal-Bench, PinchBench and OSWorld.

Code, all 25 tasks and a smaller Lite set are on GitHub under Apache 2.0. The paper was one of the most upvoted on Hugging Face’s daily papers list this week.

How does the benchmark test a delivered agent?

Each task is framed as a commission, not an exam. The builder—a coding agent such as Codex running a given model—gets a requirement in four parts (goal, interface contract and so on) plus two illustrative development cases, and can iterate against them with feedback. When development ends, its delivery is frozen.

The frozen agent then runs in an isolated, pinned runtime on six held-out cases written to cover clauses the examples leave out. Creation and Editing runs pass through hard validity gates—an agent that crashes, times out or produces an inapplicable patch scores zero—and are then graded by a model judge against a rubric derived from the requirement. Optimization runs are scored by the source benchmark’s own metric, reported as the share of the remaining gap to a perfect score that the delivery closes.

That design is closer to how people actually delegate: you describe what you want, look at a demo, and only later find out whether it works on your real inputs. It complements earlier efforts that check claims against state, such as Microsoft’s ThinkingBox, which verifies what an agent changed rather than what it said.

How did today’s coding agents do?

The main comparison uses the Codex harness with five models: GPT-5.6 Sol, GPT-5.5, DeepSeek-V4-Pro, DeepSeek-V4-Flash and Qwen3.6-35B-A3B. Creation averages were 19.1 for DeepSeek-V4-Pro, 15.1 for DeepSeek-V4-Flash, 13.7 for GPT-5.6 Sol, 5.8 for GPT-5.5 and 2.3 for Qwen3.6. Across all 50 Creation runs the mean was 11.2, the median 3.4, and 20 runs scored zero.

Editing was a little better for the strongest builders—31.6 for both GPT-5.5 and DeepSeek-V4-Pro, 23.9 for GPT-5.6 Sol—but only one run, GPT-5.5 on the Codex-editing task, reached 70. Qwen3.6 never submitted an Editing candidate, typically ending its session after announcing a plan, and DeepSeek-V4-Flash delivered on only four of ten tasks.

Optimization barely moved. GPT-5.6 Sol averaged 4.7% gap closure, DeepSeek-V4-Flash −7.1 and DeepSeek-V4-Pro −4.0, meaning those deliveries on average scored below the starter agent they were asked to improve. Of 25 runs, 13 beat their starter, nine fell below and three matched it.

Three ways builders fall short

The paper’s trajectory analysis names three recurring failure modes. The first is overfitting to visible cases. Eleven Creation runs ended development scoring 90 or more and then averaged 9.9 on held-out cases; one hard-coded its two development cases. A DeepSeek-V4-Pro Editing delivery scored 100 in development and 28.3 at acceptance. On PinchBench, two DeepSeek builders routed tasks by keyword to playbooks for the visible categories.

The second is hollow runtimes: code that reads as complete but does no work. In Creation, 157 of 270 traced held-out executions (58.1%) failed a validity gate, mostly by erroring or timing out. Static review of the source by the same judge could not tell zero-scoring deliveries from the rest (44.2 versus 45.8 on an eight-part rubric). In Editing, 28 of 33 deliveries left some required capability absent, uninvoked or incomplete.

The third is unsupported claims of completion. Six of 42 final Creation reports and three of 19 Optimization reports claimed results the builder’s own records contradicted, and delivered Editing agents misreported outcomes at run time in 13 of 33 runs. That echoes field evidence such as the Harvard study finding more code but not more finished software.

Does the harness matter?

The authors swapped harnesses while holding models fixed and found large swings on single runs. On scientific PDF translation, GPT-5.5 scored 21.8 under Codex and 42.8 under OpenCode; DeepSeek-V4-Flash scored 5.8 under Claude Code and 30.5 under DeepSeek’s own harness. No harness led for every model, so the paper reports harness–model pairings rather than ranking models alone.

Repeat builds were just as noisy. In a three-build study, GPT-5.6 Sol scored 32.8, 33.3 and 90.5 on the same DeepTutor editing task, and DeepSeek-V4.1-Flash 0.0, 33.3 and 56.8 on repository bug repair. Teams comparing tools like OpenCode 2.0 against Codex or Claude Code should treat one-off results with the same caution.

The paper also tried giving more help. Going from zero to two development cases lifted PDF translation by 34.3 points, but four cases added nothing there and lowered PPTX generation. Prescribing a design hurt PPTX; asking builders to write their own tests and run the agent end to end narrowed variance without lifting the mean. A 2.5× longer per-case time budget still left a gap of 70.3 in development versus 31.5 held-out.

How to run it yourself

The repository targets Linux x86-64 with Docker and Python 3.10 or newer; some tasks need KVM, cgroup v2 with systemd or large disks, and a `doctor` command reports what a task requires. After cloning, you set a model API key in `.env`, then use `agentswe list`, `setup`, `run <task> --builder codex` and `result` to build and score a run.

The maintainers recommend starting with AgentSWE-Lite: eight tasks built three times per builder in Codex, with DeepSeek-V4.1-Flash as both runtime model and judge, needing only a DeepSeek key. Lite scores are not comparable to the main table because the runtime model and judge differ. On Lite, DeepSeek-V4.1-Flash averaged 33.5 on Creation against 23.1 for GPT-5.6 Sol, with both at 29.3 on Editing.

What it means if you delegate agent-building

If you ask a coding agent to build you an internal agent—a report generator, a support triage bot, a data extractor—AgentSWE suggests the demo is weak evidence. Builders tune to the examples you show them, and their own completion reports are sometimes wrong.

The practical fixes follow from the failure modes: keep acceptance cases the builder never sees, run the delivered agent end to end on them, check its effects against logs or state rather than its summary, and expect variance between builds. The authors frame each failure as a training signal for the next generation of coding agents; for buyers it is also a checklist.

Limits of the study

The main comparison builds each pairing once, and Optimization metrics vary between repeat evaluations, so individual cells carry noise. Creation and Editing scores come from a model judge that shares a model family with at least one builder—GPT-5.6 Sol judges Creation and DeepSeek-V4.1-Flash judges Editing—which the authors acknowledge as a limitation.

Each task family fixes one runtime model and judge, so absolute levels are comparable within a family but not across them, and the side studies use different settings again. The builders tested are a specific set from 2026; frontier models not in the study may do better. And 25 tasks, six held-out cases each, is enough to show a pattern but not to rank close competitors with confidence.