Artificial Analysis puts frontier models in charge of a real robot arm

What did Artificial Analysis launch?

Artificial Analysis, the independent benchmarking firm known for its model leaderboards, published “Welcome to the era of embodied AI: Introducing AA-Robotics v0.1” on 14 October 2026. It calls the release a research preview that measures how well frontier language models understand, plan and act in the physical world. In version 0.1, each model gets control of one robot arm and a short objective, and the company released results for its first eight tasks.

The firm’s thesis is stated plainly. It says frontier language models have made enormous progress at controlling robot arms since the start of 2026, that the future of robotics is likely to involve strong generalist agents, and that tracking language models on robotics tasks will matter even if those models are unsuitable for production robots on their own. Its headline finding is that models in coding-agent harnesses can now manipulate a robot arm for simple tasks, but are far too slow and expensive for real use, nearly two orders of magnitude worse than human baselines.

This is the latest in a run of new Artificial Analysis evaluations in October, after its FrontierCode 1.1 leaderboard for mergeable code changes. It is the first one that leaves the screen entirely.

How does the robot-arm test work?

The setup is deliberately bare. Each model runs in an isolated container with brief instructions: how to reach the arm through a simple HTTP API, which arm it is (a Hiwonder NexArm, with no further specifications), and how to submit an attempt as complete. There is no kinematics sheet, no motion library and no demonstration. The model has to work out how joint commands translate into gripper positions by experimenting.

Vision comes from four cameras: three mounted around the table and one on the arm’s wrist. Models can write and run code in their container to calibrate, plan and send movements. GPT models ran in Codex, Claude models in Claude Code, and Gemini 4 Argon in Antigravity, all at the high effort setting. That means the benchmark measures a model plus its vendor’s agent harness, not a raw API call.

Grading happens on the end state. Once a model submits, the final scene is checked against the task’s pass criteria. A pass scores 100, and an incomplete attempt earns partial credit. Every attempt is zero-shot. The eight tasks are modeled on everyday activities: press a red button, place a toy crocodile inside a rectangle, pour a drink into a cup without spilling, tidy four blocks into a box and replace the lid, complete a shape-matching board, write “hello world” with a marker inside a black box, build the tallest freestanding foam tower, and measure a box to within 2 mm on each side.

Which models scored best, and on which tasks?

GPT-6 Astra placed first with 69.4, followed by GPT-6.1 Sol at 64.6, Gemini 4 Argon at 55.3, Claude Opus 5.5 at 48.8 and Claude Fable 5.1 at 46.3. Every model got full marks on pressing the button and on placing the crocodile. The spread opened on the harder tasks. Only Astra and Gemini 4 Argon fully poured the drink; Sol scored zero there, while Opus 5.5 and Fable 5.1 earned 25 and 30.

Sol was the only model to fully tidy the blocks into the box, where the others stalled at 40. Sol and Fable 5.1 completed the shape-matching board, while Gemini scored zero. Both GPT models wrote a legible “hello world”; Gemini and Opus got partial credit and Fable scored zero. The tower was hard for everyone: Astra’s 71 was the best result, and Sol and Fable scored zero. Measuring the box to 2 mm was the worst task overall, with no model above 17.

Grasping is where Artificial Analysis looked closest. Across six grasping tasks, GPT-6 Astra and Claude Fable 5.1 had the highest share of successful grasps at about 45 percent, Opus 5.5 hit 40, Gemini 4 Argon 36, and GPT-6.1 Sol 28, so Sol’s second place came despite the weakest grip record. On the language side, Astra also led the recent TheoPhysEval physics benchmark, and Gemini 4 Argon only reached general availability on 13 October.

How slow and expensive is it compared with a person?

Artificial Analysis says today’s models are roughly 100 times slower than a human on AA-Robotics v0.1, taking about half an hour per task compared with a human average of 20 seconds. Four of the five models average about 30 minutes; Gemini 4 Argon is slowest at 52 minutes and spends an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model. GPT-6 Astra pressed the red button in about 2.2 minutes, while Claude Fable 5.1 spent about ten minutes planning before it moved at all.

Cost is less lopsided. The firm prices human time at $37.81 an hour, the average hourly earnings on US private payrolls in preliminary September 2026 Bureau of Labor Statistics data, and says the most cost-effective model still costs about ten times as much as a person. GPT-6.1 Sol is the cheapest at $1.93 per task. GPT-6 Astra is the most expensive at $16.36, more than eight times as much as Sol, for a score four points higher. About 70 percent of the spend goes on input context, because the models keep processing camera frames of an unfamiliar scene.

Time breakdowns show where the minutes go. Models spend the opening stretch calibrating the arm and understanding the scene, then most of the remaining time planning, with short arm movements between long thinking steps. Planning took about 59 to 61 percent of attempt time for the GPT models, while Gemini 4 Argon spent 42 percent just understanding the scene, arm and movement.

What kinds of mistakes did the models make?

The most useful part of the write-up is what Artificial Analysis calls its mistake wall: short clips of things going wrong. The first category is misjudging success or when to stop. One model had the crocodile inside the rectangle at 23:22, kept adjusting for another 22 minutes, and pushed it back over the edge three times. Others reported a shape board as matching while a piece was not seated, or judged an off-center lid flush. GPT-6.1 Sol built a passing tower after about 17 minutes and then kept refining until it failed the task.

The second category is poor physical reasoning. Examples include testing pen contact on the page inside the writing box, sliding the gripper along the table into the toy and pushing it out of reach, using the lid to push the box to the limit of the arm’s reach, and not allowing for a pour stream that falls closer to the bottle as it slows. The third is plain missed grasps and drops: closing on empty air above a block, squeezing a hexagon out of the jaws, or releasing a cube too far out so it falls off a tower.

Those failures are familiar to anyone who has built robot controllers. What is new is that a general model is making them while writing its own control code on the fly, which makes the errors look less like sensor noise and more like gaps in judgment about the physical world.

Why a coding-agent robotics benchmark matters

Most robotics models are trained on robot data. Systems such as Nature Will’s Natura-1 or NEURA’s physical AI stack learn motor skills directly. AA-Robotics asks a different question: how far can a general model get if it is simply handed a robot as another tool, the way coding agents are handed a terminal? If that capability keeps improving, high-level planning and error recovery could come from general models while low-level control stays with specialized policies.

The benchmark also gives builders a cost-and-latency reference point. Half an hour and up to $16 per simple tabletop task rules out production use today, which Artificial Analysis says itself. But the price-performance gap between Astra and Sol shows the same pattern seen in software agents, where a cheaper model can sit close to the top score. Tracking that curve over time is the point of a recurring leaderboard.

Limits of a v0.1 research preview

AA-Robotics v0.1 is small. It has eight tasks, one arm model, one lab table and five models, each running in its own vendor harness, so differences partly reflect Codex, Claude Code and Antigravity rather than the models alone. Gemini 4 Argon’s activity timings are estimated, because Antigravity does not timestamp each step; Artificial Analysis interpolates them from token counts between timed arm commands. The article does not state how many attempts each model got per task, and it does not list per-task costs for Gemini or the Claude models.

Nobody outside Artificial Analysis has replicated these runs, and a single physical rig means lighting, object placement and wear can matter in ways a software benchmark avoids. Some secondary coverage has cited humanoid results; we did not find those in the primary article and have left them out. Treat the leaderboard as an early, honest baseline for zero-shot embodied reasoning, not evidence that any of these models is ready to run a robot unsupervised.