Announced 9 Oct 2026 · Sources checked
What is TheoPhysEval?
TheoPhysEval is a collection of 50 unpublished research-level problems in theoretical physics, each with a final answer that can be checked automatically. The paper describing it, arXiv:2610.16733, was submitted on 9 October 2026 in the hep-th category under a CC BY-SA 4.0 licence. Its 35 authors are physicists from institutions including the International Centre for Theoretical Sciences (ICTS) in Bengaluru, the Tata Institute of Fundamental Research, UT Austin, Kavli IPMU, Harvard, KU Leuven and the University of Wisconsin–Madison. The evaluation was coordinated by Praneeth Netrapalli of Google DeepMind and Suvrat Raju of ICTS.
The team asked experts for problems that met three tests: tied to current research, with a solution that is easy to verify but hard to guess, and not findable in the literature. Contributors were also asked not to publish their solutions soon. Many problems came from research notes: take a published calculation, change a parameter or a dimension, and ask for the new answer. The authors call these ‘turn-the-crank’ problems. A world expert could typically solve one in about a day, while experts in neighbouring fields would take much longer.
The 50 problems span 11 themes, led by quantum field theory with seven and gravitation and cosmology with six, alongside classical physics, biological physics, lattice theories, quantum phases of matter, scattering amplitudes and holography. One example asks for the Page time of a black-string geometry with two branes in four-dimensional anti-de Sitter space, adapting a five-dimensional result. Raju, an author of the original paper, says the adaptation took him about a day.
How were the agents tested?
Five off-the-shelf agents were tested, each a model paired with a common harness: GPT-6 Astra in Codex, Claude Opus 5.5 in Claude Code, Gemini 3.8 Flash in Antigravity, DeepSeek-V4.1-Flash in Codex and GLM 5.3 in Claude Code. Reasoning effort was set to maximum, except Gemini, where ‘high’ is the top setting. The runs used the Harbor framework from Terminal-Bench, the same task format Hugging Face used when it turned OpenAI proofs into FineEnvs training environments.
Each agent faced the problems on two tracks. The realistic track allowed web search, so the agent could find and read the papers a problem builds on, as a researcher would. The isolated track blocked the internet except calls to the model provider, testing what the model already knows. Every task ran on a small Google Cloud virtual machine with 2 vCPUs, 4 GB of memory and 10 GB of storage. There was no limit on tokens or cost, but tasks that did not finish within eight hours counted as no answer.
Grading compared final answers with the experts’ reference solutions. Where answers were symbolic expressions, the team added numerical versions of the questions so grading did not depend on notation. To save time and money, each agent ran the full suite only once per track. The authors say this leaves statistical uncertainty, because models give different outputs from run to run, and invite others to rerun the benchmark.
How did each agent score, and what did it cost?
On the realistic track GPT-6 Astra solved 49 of 50 problems, or 98%. Claude Opus 5.5 solved 46 (92%), Gemini 3.8 Flash 34 (68%), DeepSeek-V4.1-Flash 27 (54%) and GLM 5.3 21 (42%). On the isolated track Astra and Opus both solved 46, Gemini 31, GLM 26 and DeepSeek 25. The paper notes that the open-weight models, DeepSeek and GLM, are far from saturating the set.
The costs are as striking as the scores. Astra’s full realistic run cost $116.28 at API prices, about $2.33 a problem, which is where the paper’s ‘under USD 3’ comes from. Astra was also frugal with tokens, using 37.4 million on that track against 211 million for Opus, 607 million for Gemini and about a billion each for DeepSeek and GLM, most of it cached input. Opus was the most expensive agent, at $240.43 realistic and $275.46 isolated.
The table below lists the costs the paper reports. GLM’s run used a coding plan that cost $180 for both tracks together; at API prices the authors put it at $311.71 for the realistic track and $371.46 for the isolated one.
| Agent | Realistic track | Isolated track |
|---|---|---|
| GPT-6 Astra in Codex | 116.28 | 95.20 |
| Claude Opus 5.5 in Claude Code | 240.43 | 275.46 |
| Gemini 3.8 Flash in Antigravity | 95.01 | 90.48 |
| DeepSeek-V4.1-Flash in Codex | 37.42 | 36.23 |
| GLM 5.3 in Claude Code | 180 for both (coding plan) | API price would be 371.46 |
What did the agents get wrong?
Astra’s only realistic-track miss was an inverse problem: given a curve, find the autonomous differential equation that produced it. No agent solved it. The authors note it involves information the problem setter has and the solver does not, and some play on the wording, but kept it because researchers often face imprecise data in the same way.
The most notable detail concerns the answer key. The team first marked Astra wrong on several more problems, but on closer inspection with the experts, found subtle errors in the original reference solutions. After the experts corrected them, Astra’s answers were judged correct in every case. On the isolated track Astra failed three more problems, two of which it wrongly rejected as impossible.
Opus’s errors were also subtle. On one problem it used initial data from a cited paper instead of the data in the prompt; on another it drew a wrong conclusion about how fast an infinite series converges; and on a third it repeatedly hit its maximum output length and returned nothing on the realistic track, though it solved that problem offline. The weaker agents made mistakes an expert would spot quickly. On the Page time problem, Gemini and GLM reused five-dimensional formulas for a four-dimensional case, and offline, Gemini, GLM and DeepSeek computed geodesic lengths where minimal areas were required.
What does a 98% score not prove?
The authors are careful about the limits. Most important, scores barely changed when web search was removed, even on problems that need detailed formulas from specific papers. A human could not attempt some of them without reading the background paper. The authors conclude that leading models have effectively memorised large parts of the theoretical physics literature and can reproduce key formulas without looking them up.
They also cannot rule out leakage. A solution to one problem was published after the experiments, and some contributors used AI tools while solving their own problems, so the set does not meet a stricter standard of being untouched by AI. The authors chose not to build the set adversarially or discard problems agents could solve, aiming instead to represent real research calculations. They suggest that inverse problems could be used to make harder tests.
Most of all, the benchmark measures calculation, not discovery. The problems are, by design, ‘strictly speaking, novel, and sometimes even publishable, but not pathbreaking’. That echoes OpenAI’s recent release of 722 AI-written math manuscripts, where the hard question was which results matter. The authors also used Astra, Gemini and Opus to write the evaluation code, analyse results and help prepare the manuscript, and say so in the acknowledgements.
What changes for physicists and students?
The paper’s discussion is aimed at the field rather than at model builders. Its central claim is that frontier agents now match world experts at concrete calculations, even in specialised areas, and exceed any individual in breadth. The authors expect this to speed up discovery, and believe multi-turn work with a human checking and refining output would raise the scores of even the weakest agents.
They also see structural problems. Problems like these were often a first research project for graduate students, and the authors say it is no longer appropriate to assign them as a route to an original result. They worry that harder problems will need more compute, eroding the low cost that has made theoretical physics comparatively equal. And they argue that agents do not yet share physicists’ ‘taste’ in problems, a gap another benchmark, TasteVal, has tried to measure from a different angle.
The paper ends by turning a famous instruction to physicists, ‘shut up and calculate’, around: as taste and vision matter more, they suggest, physicists might need to ‘stop and think’. For readers, the practical point is that a few dollars of agent time now buys an expert-level calculation in many corners of physics, but checking it against a reference, and deciding whether it was worth doing, still falls to a person.
Common questions
Which AI agent did best on TheoPhysEval?
GPT-6 Astra running in OpenAI’s Codex harness solved 49 of 50 problems on the realistic track, which allows web search. Claude Opus 5.5 in Claude Code solved 46. Without internet access both solved 46. Each agent was run once per track, so the authors warn against reading small differences as firm.
Can I run the TheoPhysEval benchmark myself?
The problems are printed in Appendix A of the paper, and the authors point to a Hugging Face dataset with the problems, grader and Harbor configuration. When AI Lookout checked on 14 October the dataset was gated, so you need a Hugging Face login and approved access to download it.
Does this mean AI can now do theoretical physics research on its own?
No, according to the authors. The problems are deliberately routine adaptations of known results, not open questions. They argue agents can now do concrete calculations at expert level but still need humans to choose problems, change direction when a problem is too hard and judge what is interesting.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. The cover places the companies’ own logos, taken from public-domain or openly licensed files credited in the caption, on a plain background. It is not a product photograph, a screenshot, or an endorsement.
Our editorial standards
