FDE-Gym benchmark finds the best AI agent resolves 14 of 95 real customer delivery tasks

What FDE-Gym is

A new benchmark called FDE-Gym asks AI agents to do the job of a forward-deployed engineer: walk into a customer's running business, work out what actually needs fixing, and ship a system that improves the customer's own numbers. The answer, so far, is that they mostly cannot. In the FDE-Gym paper, posted to arXiv on 15 October 2026, the best of 13 model configurations, Anthropic's Claude Opus 5.5, averaged 0.56 on a scale where 1 is the expert reference solution, and fully resolved only 14 of 95 tasks.

The benchmark comes from DeepLethe, the company of corresponding author Dongxu Yang, with co-authors and contributors listed from Tsinghua University, Shanghai Jiao Tong University and other Chinese universities, as well as Harvard, Stanford, Cornell and University College London, and people at Huawei, Atlassian, Meta and Microsoft Research Asia. The harness and experiment runner are open source under the Apache 2.0 licence, and the project runs a public site at fde-gym.com where people can attempt the same cases.

Forward-deployed engineers are vendor staff who sit inside a customer organisation and turn a product into something that works there. The role has become a fixture of enterprise AI sales, and FDE-Gym's pitch is that existing agent tests skip the hardest part of it. Coding benchmarks hand the agent an issue and hidden tests; enterprise benchmarks hand it a predefined task. In each case, the authors write, the underlying problem is already fixed. Real deliveries start with a request that may point at the wrong problem.

How a task works

FDE-Gym has 95 tasks built from 20 customer scenarios across 18 sectors, each adapted by industry experts from a real delivery project. The scenarios include an anti-scam hold queue at a bank, a hospital triage chat, a prior-authorisation agent for a health insurer, a reconciliation tool for an asset manager, a nightly settlement job for a power grid, a newsroom flash desk, an on-call incident responder and a rollout plan for mobile robots in a warehouse. Twenty are original tasks; the other 75 are variants in which experts altered key facts of the customer's world.

The paper's opening example shows the idea. A vice president at a drilling operator wants a generative AI success story for a board meeting and asks for a daily summary email and an “Ask your wells” chat box, saying the downtime-coding system is out of scope. The real problem is that very system: a keyword script misattributes downtime to the contractor, a disputed line each month suspends that month's credit, and fixing it means finding the contract rule, checking it with a cost engineer and getting sign-off from an approver who is away that week. A delivery that does exactly what the sponsor asked can pass every functional check and still move the business metric by nothing.

Each task gives the agent a workspace with the customer's documents, data and running system, plus a cast of stakeholders it can message. Across the 20 scenarios there are more than 130 of them. Their answers are scripted rather than played by a language model, so results are reproducible, and the scripts build in limited patience, demands for evidence, people who are temporarily unavailable and contact restrictions the agent has to infer from the documents.

How deliveries are scored

Instead of pass or fail, a delivery is deployed unchanged and replayed against held-out production traffic, such as 523 bank queue calls or 190 hospital triage sessions. The score is how much it improves the customer's own success metric, normalised so that leaving the system untouched scores 0 and the expert reference solution scores 1. Messages, trial runs and waiting time with stakeholders are deducted as a contact cost. A task counts as resolved at 0.8 or above with no production incident, and scores can go below zero when a system does more harm than the one it replaced.

To stop agents gaming the grader, every task had to meet four anchors before it was admitted: the reference solution scores at least 0.95, doing nothing scores 0, building exactly what the sponsor literally asked for scores at most 0.2, and shortcuts such as overfitting to historical data, guessing round-number thresholds or handing every uncertain decision to a human score at most 0.8. The authors say the graders were tested against more than 270 shortcut and bypass strategies, and that the experts put in more than 10,000 hours.

How the 13 configurations did

The team tested 13 configurations from seven providers in the same minimal agent framework with five file and shell tools: Claude Opus 5.5, Claude Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol, GLM-5.3 Flash, DeepSeek V4.1 Flash, Gemini 3.8 Flash, Qwen 3.8 Flash, GPT-5.5, GPT-6 Luna, Mistral Small 4, and Claude Haiku 4.5 with and without extended thinking. Every task ran three times, and a run stopped at 200 turns, 150 minutes or $8 of model usage.

In the main setting, where the agent gets only the sponsor's original request, Claude Opus 5.5 led with 0.56 (95% interval 0.50 to 0.62), followed by Claude Sonnet 5.5 at 0.49, GPT-6 Astra at 0.38 and GPT-6.1 Sol at 0.37. Sonnet resolved the most tasks, 15. Five of the 13 configurations resolved none, and four averaged below zero. Even picking the best configuration for each task separately only reaches 0.70 with 30 tasks resolved, and on 16 tasks nothing reached 0.5. Opus was the best model on 11 of the 20 scenarios, but on the two hardest, the newsroom and a monthly revenue-dispute report, the best scores were 0.18 and 0.27.

Unstated rules are the biggest hurdle

The authors ran each original scenario at four instruction levels, from L0, which spells out the rules, to L3, the sponsor's raw request. With the rules given, the three leading models scored 0.73 to 0.82. Withholding the constraints while still stating the problem correctly cut their scores by 0.28 on the same tasks. Making the request vague, and then misdirected, cost only about 0.04 more.

The authors say that surprised them, since they expected problem definition to be the bottleneck. One reading they offer is that once a strong model has to hunt for constraints at all, it also discovers what the sponsor really needs. Weaker models kept losing ground at each step: GPT-5.5 fell from 0.51 at L0 to 0.06 at L3.

Getting the facts is not the same as delivering

Because the grader logs which key facts an agent extracted from stakeholders, the paper can separate finding information from building the system. The leading models asked 10 to 12 questions per task and collected 72% to 85% of the key facts, yet their systems delivered only 0.60 to 0.70 of the reference improvement. On the 23 tasks where Opus obtained every key fact, it reached 0.8 on just 10.

Asking also has a price that agents ignore. GPT-6 Astra's systems hit 0.8 on 36 tasks before the contact charge and only 9 after it; for Opus the figures were 40 and 14. The agents are not told questions cost anything, and the paper notes that they ask as if questions were free. At the other extreme, Mistral Small 4 asked 0.7 questions per task and found 6% of the facts.

Spending did not track results. GPT-6.1 Sol used $0.24 and 25 turns per run to score 0.37, while Gemini 3.8 Flash spent $1.54 and 121 turns for 0.19. Qwen 3.8 Flash worked longest and submitted within the limits on only 11 of 152 combinations, which the authors read as a sign that knowing when the job is finished is part of the skill.

Weak agents leave customers worse off

The sharpest finding for buyers is that a weak agent is not just a cheaper strong one. Claude Haiku 4.5 scored below the do-nothing baseline on 79 of 95 tasks and GPT-6 Luna on 63, compared with 5 for Opus. Before any contact charge, Mistral Small 4's systems were below zero on 51 tasks, for example paying out claims they should refuse or waking the wrong engineer. Similar warnings about agents acting on their own initiative surfaced in a recent train-refund experiment with six agents, and in the AgentSWE benchmark, where coding agents rarely built what was asked.

The authors also compare FDE-Gym with public results from Vals AI. Models that solve up to 88% of Terminal-Bench 2.1, 90% of Vibe Code Bench and 61% of Finance Agent tasks score 0.19 to 0.56 here. They stress that success rates and normalised uplift are different quantities, but conclude that component benchmarks are a weak guide to what a model will deliver in an engagement.

How to try it

The open-source repository runs the site, a runner, a PostgreSQL database and an object store with Docker Compose, and the harness can work a case from the command line for a person or an agent. A cheap model reached through OpenRouter maps questions to scripted answers. Three complete cases, a campus booking assistant, an on-call incident responder and a warehouse robot rollout, are public in a Hugging Face dataset under CC BY-NC 4.0. The 20 original tasks are available on request, while the 75 variants and all reference solutions are held out for the leaderboard. The maintainers ask people not to train on the cases.

Limitations

All results are the authors' own and have not been independently reproduced. Every model ran in one minimal framework, so the scores describe the models there, not products such as Claude Code or Codex, and the paper says a stronger framework could change both scores and ranking. Stakeholders are scripted, and contact costs are expert estimates rather than measurements.

There is no human baseline yet; one is being collected on the site and will appear in a later version. The instruction-level comparisons rest on only 20 tasks. The scenarios lean toward regulated operations, and because they are adapted from real deliveries, the authors cannot rule out related material in training data. The repository describes itself as early, with a database schema that still changes between versions.