TasteVal benchmark finds Claude Opus 5.5 out-experiments human AI researchers

What is TasteVal?

Oliver Jaffe and Dane Sherburn of P-Zero Research published TasteVal on arXiv on 5 October 2026 and revised it on 6 October, alongside a project write-up. The paper defines research taste as the ability to pick interesting problems, design experiments and interpret results, and then measures only the middle and last parts: given a fixed problem, how well does a model choose experiments and learn from them?

The authors turn that into a number. A model that reaches the same score as the human expert baseline using half the serial GPU time has twice the experimental taste. They call this the compute multiplier, and argue it matters for forecasts because, once coding is automated, experiment compute becomes the bottleneck on AI progress.

TasteVal has eight new tasks built for the benchmark, spanning pre-training data curation, pre-training and fine-tuning language models, preference modelling and robustness to adversarial prompts. To keep them uncontaminated, the authors are not releasing the tasks.

How does the benchmark separate taste from coding?

Each run splits the work between two agents. The model being tested is the Researcher: it reads the task, the training and validation data, and experiment reports, and writes experiment descriptions in plain language. It cannot run Python or see the code. A fixed Coder, Opus 4.8 inside Claude Code, implements each experiment on a single H100, logs metrics and writes a report.

Two model-based monitors police the boundary, one checking experiment descriptions before the Coder sees them and one checking reports before the Researcher reads them, and a Liaison answers the Researcher’s questions by reading the Coder’s transcript. The Coder can score against the test set but is told never to leak it; the Researcher only ever sees validation scores.

Every attempt gets 40 GPU-hours of busy H100 time or 120 hours of wall-clock time, whichever runs out first. The Researcher’s own tokens do not count against the GPU budget. The human baseline uses the same setup: 24 experts who have recently worked at organisations including OpenAI, Google DeepMind, NVIDIA, Microsoft Research and Oxford, at least two per task, with the best attempt per task used as the bar.

To check the split is not hobbling models, the authors also ran Opus 4.8 end to end, writing and running its own code. It scored 0.80 times the baseline versus 0.84 in the split setup, which they call statistically indistinguishable.

What did the 20 models score?

The team evaluated 20 models released between 2023 and 2026, with six seeds per model per task. GPT-4 scored a 0.03 times multiplier and o1 scored 0.07, meaning the expert baseline needed about 7% of o1’s compute to match its final score. Opus 4.8 sat at 0.84. Opus 5.5 at maximum reasoning reached 2.30 times, with a 95% confidence interval of 1.15 to 4.37 that stays above the expert line.

The striking result is the trend. A single exponential fits poorly; a statistical break test puts a kink at 11 December 2025, the release of GPT-5.2. Before it, frontier compute multipliers doubled every 14.0 months. Since then, they have doubled every 3.0 months, with a 95% interval of 1.7 to 5.0 months. Measured by final score regardless of compute, there is no break: a steady doubling every 14.6 months.

Cost is part of the story. The paper puts Opus 5.5’s average run at $282 in API spend and GPU time, against $9,123 for an expert run including salary. The authors note that matching budgets in dollars rather than GPU-hours could make agents look stronger still, and that a minor prompt-caching change cut Researcher API costs about fivefold without changing scores.

They also report behaviour. Recent models keep improving late in a run: 43% of Fable 5.1 runs improved their selected submission after 30 GPU-hours, against 21% for GPT-5. But models were worse than experts at judging significance; the strongest humans measured run-to-run noise first and discarded gains smaller than it, while Fable 5.1 rarely re-ran with a new seed.

Are the models proposing new ideas?

To test whether models were inventing anything, the authors had Fable 5.0 label 540 official model submissions as Tuned, Composed, Modified or Invented against the literature. None qualified as Invented, and none of about 1,600 experiments inside top-decile runs did either. Composed submissions, which combine published pieces, were the majority above a 0.5 times multiplier, and Modified ones were rare before GPT-5.6 Sol.

Human experts looked similar: 20% Tuned, 72% Composed and 8% Modified. The authors’ reading is that progress so far comes from shallow and moderate optimisation, better tuning and adaptation, not new ideas, and that the trend could slow if further gains need deeper invention.

Two other experiments cut both ways. Spending more inference helped: Opus 5.0 in a scaffold that forced at least 200,000 tokens of thinking between experiments matched Opus 5.5 statistically, at 42% higher total cost, and tended to pre-register predictions and kill criteria. Hiding the scoring metric on four tasks roughly halved multipliers, from 1.12 to 0.54 for Opus 5.0 and from 0.14 to 0.057 for Opus 4.5, though with four tasks neither drop was statistically significant.

Why is Jack Clark now saying 2027?

In Import AI 476 on 12 October, Anthropic co-founder Jack Clark highlighted TasteVal and wrote that creativity had been the variable he was watching for recursive self-improvement. “I now think it’s more likely we’ll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028,” he wrote. That update is what pushed the paper into wide discussion on 13 October.

The paper itself goes further with forecasts, but labels them naive extrapolations. Plugging its post-break rate into the AI Futures Model, under stated assumptions, raises the probability of a taste-only singularity from 51% to 88% of simulations and moves median ASI arrival from 2030.5 to 2028.9.

Those numbers depend on assumptions the authors spell out: that the post-2025 rate continues, that unmeasured parts of taste such as choosing problems improve at the same rate, and that a single-H100 benchmark transfers to frontier labs’ parallel, messy research. Readers should weigh Clark’s update with that in mind, and note that the top-scoring model comes from his own company.

What the benchmark does not measure

The authors list limitations that could overstate progress. Tasks have a clean validation set and fast, low-noise scoring, unlike most frontier work. Compute is tiny next to real training runs. TasteVal does not test picking problems worth solving. And the tasks are private, so outside groups cannot rerun them; the evidence for now is the paper and its transcripts.

Other choices could understate it: limited elicitation of both agents, a Coder that is not the newest model, and agents spending a thirtieth of what experts cost.

The authors also say they trust the trend more than any single model’s number, because per-task multipliers often differ by orders of magnitude and resampling tasks dominates the uncertainty. The break date itself is fitted to the data, with models dated by when each lab first made them generally available.

How TasteVal fits the AI research-agent picture

TasteVal sits in a fast-growing shelf of benchmarks for AI doing AI research. Hillclimb’s DeepRSI runs agents end to end for 48 hours, while JANUS tries to make research agents forecast experiments before spending compute, which is exactly the skill TasteVal rewards.

Methodology fights around METR’s time-horizon trend are a useful warning: trend breaks fitted to a handful of frontier points can move when new data or methods arrive. Work such as Sakana’s MASS paper on recursive self-improvement and OpenAI’s AI-written math manuscripts shows why labs and forecasters are watching this so closely.

For practitioners, the practical lesson is narrower. If you run experiments with agents, measure how much compute they need to hit your target, not just their final score, and copy what the best humans did: estimate noise first and refuse to believe single-run gains.