Breken launches a production simulator that caught 76 of 235 shipped bugs in its own test

What Breken launched

Breken, a new software-testing startup, launched its product of the same name on 21 October 2026 and described it as the start of a new job it calls production engineering. Breken rebuilds a customer's whole application on its own machines, grows a realistic production history around it, and then searches that simulated world for the failures a real user would otherwise find first.

Alongside the launch the company published ESCAPES, a benchmark built from 327 real bugs that slipped past code review and tests in 33 open-source projects and were later reported by users. On Breken's own run, its system caught 76 of the 235 bugs that were already in the code on 1 July 2026, about 32 percent. The best AI coding agent it compared against, Claude Fable 5.1 working through Claude Code, caught 37.

Those numbers come from Breken, which designed the benchmark and ran every system on it. The dataset, prompts and labels are public, so others can check them, but no outside group has reproduced the results yet.

How a production before production works

Breken's argument starts from a simple observation: tests check what a team thought of, and production checks everything else. As AICoder summarises the pitch, today a product's users run that second test for its makers.

To run it earlier, Breken says it reconstructs the entire system in an isolated environment. Outside services such as payments, mail, databases and queues are answered inside that sandbox rather than mocked, and nothing in it is reachable from the internet. A simulated clock runs at 4,000 times normal speed, so months of accounts, records, pending jobs and half-finished sessions build up the way they would in a live product.

Into that world it releases synthetic users and AI agents. The company describes impatient admins doing bulk edits, a new user with the same document open in two browser tabs, and an agent issuing refunds from an inbox, each with its own intent, habits and limited patience, and each reproducible.

Every property the software is supposed to hold, such as a retry charging a customer only once, becomes a hypothesis. Breken tests it by changing one condition at a time while holding everything else fixed: a retry, a response lost after a database commit, a jump in the clock or a deploy in the middle of a job. A failure only counts if it reproduces every time against a control run that never fails. When it does, the customer gets the exact state, a trace and a draft pull request with a proposed fix.

The ESCAPES benchmark

ESCAPES is Breken's attempt to measure that layer. Each of the 327 bugs was reported by a real user on a project's public tracker and fixed by a merged pull request between 1 July and 19 October 2026. The 33 projects range from web apps and API servers to command-line tools and SDKs, including the GitHub CLI, Homebrew, rclone, Streamlit, Gradio, Etherpad, Shopware and Vercel's AI SDK.

Breken chose 1 July because it says that is the first day after the training cutoff of every model it tested, so none of them could have seen the bugs or their fixes. Each system got one scan of each project's code as of that date and was asked to report the bugs users would hit. Of the 327 bugs, 235 were already in the code then, and those are the ones scored.

The coding agents, Claude Fable 5.1 and Claude Opus 5.5 in Claude Code and GPT-6 Astra in Codex CLI, were given the same prompt, the same spend cap and a limit of ten findings per project. Semgrep's default rules served as a floor. A catch counts when a finding describes the failure the user later reported. Where possible it is proven by running a reproduction that fails before the fix and passes after; the rest are graded by GPT-6 Sol as a judge and then checked by people.

On that basis Breken reports 76 catches with 20 false alarms, 61 of the catches proven by running. Claude Fable 5.1 caught 37 with 59 false alarms, Claude Opus 5.5 caught 33 with 69, GPT-6 Astra caught 28 with 86, and Semgrep caught 9 with 112. Breken puts its own precision at 85 percent against 46 percent for Fable 5.1, and says its catches came a median 44 days before the user's report.

Where running beats reading

The more interesting finding is not the headline score but where the gap appears. Breken labelled each bug by what it takes to see it. For the 69 bugs visible by reading a single file, Claude Fable 5.1 and Breken caught about the same share, 33 and 32 percent.

The difference is in the other 166, the bugs that need the product running, a real data shape, a second user, an outside service failing or time passing. There, Breken says, Fable 5.1 found 14 and Breken found 54. By the company's count, 71 percent of all the bugs only appear when the product runs, and they sat in the code for a median 152 days before a user reported them.

That fits a wider pattern in AI-assisted development. Tools such as Claude Code Review inspect a diff before it merges, and Anthropic has said its own team doubled merged pull requests while AI review cut reverts. Breken argues those tools and its own sit at different layers: review sees the change, while a simulator sees the running product. Its benchmark page says plainly that the two are not alternatives.

Three catches and one miss

Breken illustrates the results with four cases from the benchmark. In an e-commerce platform, refunding an order after a partial capture refunded the full authorised amount; Breken ran checkout with a partial capture, refunded, and saw the ledger show the whole sum, 47 days before a user reported it.

In a storage sync tool, a transfer cut off by a dropped connection was reported as complete and the half-copied file stayed marked as synced. Breken cut the connection mid-transfer, synced again and found the checksums differed, 63 days ahead of the user. In a collaborative editor, two simulated people working on one paragraph during a brief disconnect lost an edit in three of three runs.

The miss is instructive too. In an AI API gateway, streamed responses with tool calls were billed twice. Breken tested streaming and tool calls, but never both together on a billed account, because that combination was cut by the run budget. The company says it now tries that combination first on every AI gateway and keeps the miss in the record.

How to try it and what it costs

Developers sign in at breken.ai with GitHub and choose which repositories to connect. According to AICoder's launch report, usage is prepaid, with a minimum top-up of $25 and preset amounts of $50, $100 and $250.

The obvious fit is software with a lot of state and money on the line: payments, billing, sync, collaboration and anything with renewals or expiry dates. A sensible trial is to point it at a service with a known history of incidents and compare what it finds, and how many false alarms it raises, against bugs the team already knows about.

The launch arrives as AI-written code raises the volume of changes reviewers must handle. PostgreSQL, for example, pushed back its version 19 release after AI-assisted review produced a pile of bug reports, and Cohere is buying the AI bug hunter Aisle for security work. Finding bugs is becoming a crowded category.

Limits and open questions

Breken is open about the main caveat: it built the benchmark and ran every system, including its rivals. The prompt, caps, judge prompts and labels are public, and a vendor's own submission replaces Breken's run of its system, but until Anthropic, OpenAI or an independent lab submits, the comparison is Breken's.

The setup also favours a tool that runs software over agents that mostly read it. The coding agents could build and run the projects, but were capped at ten findings and the same budget. Each result is a single run, only bugs linked to a fix are included, only two bugs involve several users, and no browser-agent products were tested. Cost and wall time were recorded but not scored, so the price of a 32 percent catch rate is not yet known.

Breken says it will publish a new version of ESCAPES every quarter from fixes merged after the newest model cutoff. Outside submissions, independent reruns and real customer results will show whether a simulated production really catches what users would.