Xplore's RWE-bench tests AI agents on real engineering work, and the best model passes 37 percent

What RWE-bench measures

Xplore released RWE-bench (Real World Engineering Bench) on 27 October 2026. It contains 20 tasks across mechanical, electrical and aerospace engineering, each written by an Xplore engineer who works in that domain, along with a reference design built with the same tools the agents get.

The tasks read like real engineering work rather than a quiz: design a one-piece steel crankshaft for a three-cylinder diesel engine with oil drillings that must take full-load firing at 2,400 rpm and last millions of cycles; route the DDR3L memory interface on a production FPGA board with 71 nets out of a 484-ball BGA; and run CFD around a high-pressure turbine vane with film cooling, from building the mesh to a converged solution.

The results

Across the leaderboard dated 28 October, OpenAI's GPT-6 Astra passes an average of 37% of its runs, with a 95% credible interval of 30.7-43.8%. Claude Opus 5.5 follows at 29% and GPT-6.1 Sol at 25%. Five of eleven models scored zero: DeepSeek V4.1 Flash, Qwen3.8 Max, GLM-5.3, Grok 4.7 and Meta's Muse Wonder.

The intervals of the top two overlap, so Astra's lead over Opus 5.5 is not statistically settled. Four of the twenty tasks defeated every model across nearly 1,100 total runs. Xplore's own framing: 'AI now beats almost every human at math and coding, but cannot reliably solve physical engineering problems.'

Grading and harness caveats

Every run is graded automatically by a deterministic verifier, with no LLM judge and no human grading. A program checks the submitted design against every requirement, including by running physics solvers — the crankshaft gets checked for being a valid solid, clearances, mass, balance, proof stress, fatigue, stiffness and bearing loads.

Notably, each model runs inside its own vendor's agent: Codex for OpenAI, Claude Code for Anthropic, Gemini CLI for Google. That means the score measures the model and its harness together, not the model alone. The data file is provisional, coming from an October 25 export made while some runs were still in progress, and effort settings vary between models. The benchmark page offers neither the verification programs nor the reference solutions without contacting the team, and no independent replication had been published as of 28 October.

The human question

Xplore does not publish a human baseline. The page says passing any of these tasks takes an expert engineer trained in computer-aided engineering software, but it offers no measured pass rate or time for engineers. A Reddit post on 28 October claimed most people would pass none of them, and independent commentators generally agree a non-engineer would fail all 20.

That context matters: a 37% pass rate looks low until you consider that the tasks are chosen because they stump humans too. Xplore's purpose is not just evaluation but training — it builds reinforcement-learning environments for physical engineering, and Lightspeed's support for the launch emphasized that physical engineering still lacks the evals and verifiable rewards that domains like code and math share.

What it means

RWE-bench joins a small but growing set of physical-domain benchmarks, including Google Cloud AI Research's PLE (85 tasks, top agent score 50.6%). The low scores across both benchmarks suggest that physical engineering — where geometry, stress, heat and impedance constraints must all hold simultaneously — remains a hard frontier for AI agents.

The deterministic physics-based verifier design is the key innovation: it gives RL training a dense, trustworthy reward signal, which is exactly what model providers need to improve. The benchmark is small (20 tasks) and the top-2 gap is not statistically significant, but the signal is consistent: frontier models are nowhere near professional-grade engineering yet.

What to watch

Watch whether Xplore opens the verification programs and reference solutions, which would let others replicate results.

Watch whether model makers use RWE-bench as a training target and move scores meaningfully in the next few months.

Watch for a measured human baseline, which would give the 37% figure its proper context.

The three sample tasks

Crankshaft design: the agent designs a one-piece steel crankshaft for a three-cylinder diesel engine in CAD, with oil drillings. It must fit given bearings, flywheel and front drive, clear the connecting rods as they turn, and balance counterweights without exceeding a mass limit. Under full-load firing at 2,400 rpm the shaft must stay within stress, stiffness and bearing-load limits and last millions of cycles without fatigue failure.

DDR3 memory routing: the agent routes a 32-bit fly-by memory bus from a Zynq-7000 to two memory chips on a production FPGA board. That means 71 nets out of a 484-ball BGA on an eight-layer board, around nearly 300 parts already placed, with every data line within its skew budget of its strobe and every track holding target impedance.

Turbine vane CFD: the agent predicts heat transfer around a high-pressure turbine vane that sits behind the combustor, where gas is hotter than the metal can survive. It must build and mesh the flow passage, run CFD to convergence, and report heat-transfer coefficient along the vane surface and total heat load.

Attempts and effort settings

Each model runs in its native harness at maximum reasoning effort, with five attempts per task and up to 24 hours per attempt; the average run took 3.4 hours across all models. Xplore also reports a pass-rate-over-attempts view in which a task counts as solved if any run passes: with five attempts, Claude Opus 5.5 passes 70% of tasks, ahead of GPT-6 Astra at 60% and GPT-6.1 Sol at 50%.

That measure paints a different picture: on best-of-five, the top three all clear half the tasks. The single-run average is the stricter metric and the one Xplore leads with, but the gap between the two reveals how much of these results is variance between attempts on very hard tasks.

Comparisons with PLE

RWE-bench is not the only physical-engineering evaluation. Google Cloud AI Research's PLE benchmark has 85 tasks across electrical, civil, structural and mechanical engineering, with the top agent scoring 50.6% mean against executable graders with reference deliverables.

Where PLE targets evaluation of existing frontier models against professional deliverable standards, RWE-bench is positioned as both an evaluation tool and a source of training signal. Xplore builds reinforcement-learning environments for physical engineering, and its distributed verifiers are designed to serve as the 'verifiable rewards' that agents need to improve in physical domains — the gap Lightspeed highlighted in its support for the launch.