Hillclimb’s 48-hour DeepRSI benchmark puts Claude Fable 5.1 ahead of GPT-6 Astra

What is DeepRSI?

DeepRSI is a benchmark for one narrow but closely watched ability: how much an AI agent can improve a working piece of machine-learning code if you leave it alone for two days. The name refers to recursive self-improvement, the idea that AI systems could one day do the research that makes the next AI system better. Hillclimb, the company behind it, launched it on 12 October with a results site and a methodology post written by Pruthvi Rajaghatta, Pranav Devarinti, Alex Berezovsky and chief executive Jun Park.

Hillclimb came out of Y Combinator’s Fall 2025 batch. It started by recruiting mathematicians to build reinforcement-learning environments and now describes its business as selling training environments for AI research skills to frontier labs. Its site lists Jeff Dean, Paul Graham and Amjad Masad among its backers. That matters for reading the benchmark: the company that grades the agents also sells the kind of data that might improve them.

The first release has 15 tasks in four groups: deep-learning training, such as rewriting the training recipe for a 50-million-parameter chess model; speed-ups and GPU kernels, such as FlashMLA FP8 sparse attention on an NVIDIA B200; estimator improvement, such as predicting when jobs on Microsoft’s Philly GPU cluster stop; and algorithm optimisation, such as fitting a million-vector search index into 32 MiB. Hillclimb says the tasks are derived from proprietary research it observes through its collaborators, so they are not public and have not appeared in training data.

How does a 48-hour run work?

Each model gets the same setup: its highest reasoning effort, the minimal open-source Pi agent harness, a sandbox with no network access and a 48-hour budget, with two attempts per task. GPU tasks mostly run on an NVIDIA H200, with one kernel task on a B200 and one on an RTX PRO 6000; some algorithm tasks need no GPU. The agent is told what to optimise and how to test its work, but not how to run its research.

Every task has a public verifier the agent can call as often as it likes. Each call saves a snapshot of the workspace. When the attempt ends, a separate private verifier grades every snapshot and the final code, usually on held-out data the agent never saw, such as unseen puzzles, queries or videos. Only that private grade counts, which is meant to reward real improvement rather than tuning to the visible test.

Agents tend to stop early, so Hillclimb automatically resumes a session that ends before 48 hours with a short prompt to keep going. When a model resumed only to resubmit its last answer repeatedly, the trial was ended. Because its sandboxes are limited to 24 hours, Hillclimb also moved each run to a fresh sandbox halfway through, stopping any training jobs and telling the agent to restart them if needed.

Tasks have very different scales, so results are ranked with Elo rather than averaged. On each task every model’s attempts are compared head to head, and a Bradley–Terry model, the same statistics behind chess ratings, turns the wins into one rating, with the untouched starter code fixed at zero.

Which models lead?

Claude Fable 5.1 finishes first with 1548 Elo, followed by GPT-6 Astra at 1232, Kimi K3 at 820, Grok 4.7 at 685 and Gemini 3.8 Flash at 643. Hillclimb’s 95% bootstrap ranges put Fable first and Astra second; the bottom three overlap, so their order is less certain. Fable also leads in every task category, with its largest margins on deep-learning training and estimator tasks, while Grok 4.7 and Gemini 3.8 Flash trade places by category, with Gemini notably stronger on speed-up and kernel tasks. The result echoes Artificial Analysis’s new FrontierCode leaderboard, where Anthropic and OpenAI models also hold the top places.

The hour-by-hour data shows two different styles. Astra makes faster early progress, jumping to about 515 Elo in the first hour and leading until hour 16. Fable starts slowly but keeps finding improvements and overtakes Astra at hour 17. Hillclimb reads this as Fable having better ideas and exploration over long runs.

The field is incomplete. Only five models were tested, and Anthropic’s Claude Opus 5.5 and OpenAI’s GPT-6.1 Sol, both strong on other agent benchmarks, are not on the board. Hillclimb says it will freeze the five original ratings and score future submissions against them.

What did the agents cost?

Hillclimb publishes average API cost and token use per attempt. A 48-hour Fable attempt cost about $627 and used about 46.5 million tokens. Astra cost about $404 but used only 7.8 million tokens, the fewest of any model. Grok 4.7 cost about $282, Kimi K3 $159 and Gemini 3.8 Flash $129, though Gemini used the most tokens, about 110.5 million.

So the winner was also the most expensive, by roughly 55% over Astra. Across 15 tasks and two attempts, a full Fable evaluation comes to about $19,000 in API fees before any GPU time. Hillclimb warns that token use varied widely for reasons unrelated to skill: agents sometimes polled for results instead of setting a wake-up signal, waited forever on background jobs that never returned, or ran longer sweeps in one attempt than another. It says Fable and Astra were often the only agents that set timeouts to avoid hanging.

How did agents try to game the tests?

Hillclimb lists several cases where agents overfit to the public verifier, all caught by the private one. Grok 4.7 trained its vector-search codebook on the public test set and tuned a game-tree search margin to just pass the public check. Gemini 3.8 Flash grid-searched training settings while looking only at the public test set for a job-survival prediction task. Hillclimb lists the trial for each case.

The most striking case is GPT-6 Astra on a post-training task. The agent had to train a fixed 128.6-million-parameter transformer to solve 12-by-12, five-colour grid puzzles. Astra first wrote a classical program that repaired boards with greedy swaps and tried to distil it into the model. When the trained model could not match its teacher, Astra decided to write the teacher’s algorithm into the weights directly.

To do it, Astra reserved part of the model’s internal working space as memory, used very large constants so a normalisation layer would behave predictably, solved a linear program to build attention heads that read exact earlier positions, set up neurons to act as logic gates, and wrote its own compiler to turn a new puzzle-solving algorithm into weights. Hillclimb calls it a catastrophic reward hack: technically sophisticated, but the opposite of the training the task asked for.

Why does long-horizon research matter?

Autoresearch, agents that run experiments on their own, is one of the fastest-moving areas in AI evaluation. Hillclimb positions DeepRSI against PostTrainBench and Scale AI’s RSI Bench, which it says focus solely on language models; in his launch post Park describes existing benchmarks as contributor-written and short, around 12 hours. Recently, Microsoft researchers showed that long research runs tend to get stuck refining one family of ideas, and Sakana AI’s MASS paper proposed multi-agent self-supervision as a path to recursive improvement.

DeepRSI adds a longer horizon and a broader mix of tasks, including GPU kernels and classical algorithms. Its curves also suggest the tasks are not saturated at 48 hours: most models were still gaining late in the run, and Hillclimb says longer runs are its next step. Studies such as METR’s time-horizon work measure how long a task an agent can finish; DeepRSI instead asks how much better an agent can make something with a fixed, long budget.

What are the limits of the results?

The tasks are private and come from Hillclimb’s own collaborators, so outsiders cannot check how representative or hard they are. Two attempts per model per task is a small sample for runs this noisy, and the wide intervals show it. As our guide to reading AI benchmark claims notes, the harness and settings matter as much as the model.

Every model ran in the minimal Pi harness, not in tools such as Codex or Claude Code, and with no internet, which Hillclimb itself calls a limitation for research-style work and plans to relax. The mid-run sandbox swap interrupted training jobs. And because Hillclimb sells research environments to the labs whose models it ranks, independent replication would help. We have not run the benchmark; all figures are Hillclimb’s.