Artificial Analysis launches a FrontierCode leaderboard, and Claude Opus 5.5 leads at 52%

What did Artificial Analysis launch?

Artificial Analysis, the independent benchmarking firm best known for its Intelligence Index, published a new leaderboard on 12 October for FrontierCode 1.1, a coding benchmark built by Cognition, the company behind the Devin coding agent. Each task asks an agent to make a change to a real open-source repository. Instead of only checking whether tests pass, the benchmark asks whether a maintainer of that repository would accept the change.

Artificial Analysis runs the 100-task Main set, out of 150 tasks in total, three times per task, with every model in the same harness: Stirrup, its own open-source agent harness, capped at 250 turns per attempt. The tasks themselves are private to avoid contamination. Twenty-two models had been run when we checked on 13 October.

FrontierCode launches as a separate leaderboard and is not part of the Artificial Analysis Intelligence Index, the composite score Meta quoted for its new Muse Wonder models on the same day. Artificial Analysis says it plans to add FrontierCode to its Coding Agents leaderboard soon.

How does FrontierCode decide what is mergeable?

Cognition built FrontierCode with the maintainers of 36 open-source repositories, including projects such as Celery, Budibase, Uppy and Mattermost. In its June launch post, Cognition said more than 20 maintainers each spent over 40 hours per task, writing a short, human-style task description plus the project’s usual testing, lint and style guidelines, and then deciding what “mergeable” means in their own codebase.

Each attempt is graded with six kinds of check. Classical tests are injected and run; command checks run the build or linter; reverse-classical checks run the agent’s own new tests against the original code to make sure they actually fail there; adaptive tests let a model adjust reference tests to fit a valid alternative design; scope checks limit which files may change and how big the diff can be; and an LLM reviewer judges the diff against the maintainer’s written criteria. Artificial Analysis uses GPT-6 Luna as that judge.

Some checks are blockers, the things a maintainer would treat as a hard stop in review. Fail one and the attempt scores zero. Pass them all and the score lands between 70% and 100%, depending on softer qualities such as style and readability. That is why the leaderboard reports two numbers: the score, and the pass rate, meaning the share of attempts that clear every blocker.

Which models lead, and by how much?

Claude Opus 5.5, at its maximum reasoning setting with Anthropic’s default fallback, leads with a 52.2% score and a 58% pass rate. GPT-6.1 Sol (49.1%) and Claude Fable 5.1 (49.0%) follow, then GPT-6 Astra (47.0%) and Claude Sonnet 5.5 (45.8%). In Artificial Analysis’s words, leading models still fail more than 40% of the time.

The pack is tight. Nine models from six labs score between 43% and 52%. Xiaomi’s MiMo-V2.6-Pro is the best open-weights model at 45.4%, with DeepSeek V4.1 Flash and Kimi K3 at about 44%. Smaller and cheaper models are not far behind: Claude Haiku 5.5 scores 41.8% and GPT-6 Luna 41.9%. At the bottom are MiniMax-M3 at 23%, Thinking Machines’ Inkling at 16% and Meta’s open Muse Glimmer at 14%, which passes only 16% of attempts.

Two details show how hard the hardest tasks are. Nine of the 100 tasks were passed by no model on any attempt, and no task was passed by every model every time. Meta’s new Muse Wonder models, released the same day, were not yet on the board when we checked.

What does a point of FrontierCode cost?

Cost per task, at list API prices, spans more than a hundredfold, from $0.09 for GPT-6 Luna to $12.00 for Claude Fable 5.1. Artificial Analysis identifies four models on the cost frontier, where no other model is both cheaper and better: GPT-6 Luna (42%, $0.09), MiMo-V2.6-Pro (45%, $0.18), GPT-6.1 Sol (49%, $0.81) and Claude Opus 5.5 (52%, $10.15).

The last step is expensive. Moving from GPT-6.1 Sol to Opus adds three points for more than 12 times the cost per task. GPT-6.1 Sol also scores two points above the bigger GPT-6 Astra at about a fifth of the cost. Token use explains much of the gap: GPT-6 Astra and GPT-6.1 Sol produce about 27,000 and 33,000 output tokens per task, against about 157,000 for Claude Opus 5.5 and 236,000, the most of any model, for Claude Sonnet 5.5. Sonnet still costs less than Opus per task, $8.26 against $10.15, because its tokens are half the price.

Time varies too. Depending on token count and output speed, a task takes from about three minutes for Inkling to about 46 minutes for Qwen3.8 Max.

Why do internet rules matter for coding benchmarks?

FrontierCode tasks come from real pull requests, so a fix often already exists somewhere online, in a later version of the repository, a mirror or a package registry. Cognition says recent models had become good enough at searching that some were simply finding the answer. In version 1.1, released in July, it chose not to switch the internet off, because some tasks need documentation and because search is now part of how agents work.

Instead, agents get a prompt explaining fair use, such as reading docs, versus unfair use, such as opening the upstream pull request, and a scanner zeroes any run that touches solution-bearing sources. Cognition says the prompt alone pushed unfair use below 1% for every model it tested. It also tried a blocklist, which grew to about 1,200 domains as agents kept finding workarounds. Version 1.1 relaxed 75 overly strict blocker criteria and dropped the 50-task Diamond subset as too noisy.

What should developers and buyers take from it?

The headline lesson is that passing tests is no longer the bar. Earlier benchmarks such as SWE-bench check functional correctness, and METR found in March that many SWE-bench-passing patches would not be merged. FrontierCode puts a number on that gap: even the best model fails a maintainer’s hard requirements in about four of every ten attempts. That fits the finding of a Harvard study of 718 firms that coding agents add code faster than teams can review it.

For teams choosing models, the cost data may matter more than the ranking. If your workload looks like FrontierCode, GPT-6.1 Sol gets within three points of the leader for under a dollar a task, and MiMo-V2.6-Pro offers an open-weights option close behind. The extra points from Opus are real but costly, and output tokens also mean review time and latency.

What are the limits of the results?

The tasks are private, so nobody outside Cognition and its partners can inspect them, and quality judgments rest partly on an LLM judge, which can be wrong. Every model runs in Artificial Analysis’s Stirrup harness; vendors’ own agents, such as Claude Code or Codex, may score differently, and Cognition’s own leaderboard, run with its methodology, reports separate numbers.

Three attempts per task leave some noise, especially near the top, where several models are within a point or two. Results reflect the reasoning settings and prices on 12 October and will shift as models and prices change. Finally, 100 tasks from 36 repositories is a careful sample, not all of software engineering; it says little about greenfield apps, large refactors or work in closed codebases.