Announced 9 Oct 2026 · Sources checked
What is JANUS?
Researchers at the Institute of Computing Technology, Chinese Academy of Sciences, posted JANUS on arXiv on 9 October 2026 (arXiv:2610.13194). The name nods at the two-faced Roman god: one face forecasts future experiment outcomes; the other looks back to absorb ground-truth feedback.
The problem they target is familiar to anyone watching AutoResearch agents chew GPU hours. Systems in the Generate–Execute–Feedback style propose a change, run training or evaluation, read the log, and try again. When each try can take hours, combinatorial branching wastes budget on dead ends.
JANUS’s answer is a research world model that sits inside an evolving directed acyclic graph of proposals. Before an executor spends budget, the world model estimates whether a candidate is worth running. After the run, JANUS updates hierarchical skills so both the proposal generator and the predictor improve together. A project page is listed at github.com/bilemond/JANUS.
How does the Predict–Verify–Evolve loop work?
Each iteration has four moves. Expansion: an orchestrator samples candidate proposals by modifying, extending or combining nodes already on the research graph. Prediction and refinement: the world model scores candidates and may debate with the orchestrator to revise weak proposals before anything runs.
Implementation and execution: a code executor turns the chosen proposal into code, evaluates it and deducts cost from the remaining budget. Co-evolution: JANUS reflects on what worked, summarising raw trajectories into intra-task and cross-task skills for the research agent, prediction and exploration skills for the world model, and coordination skills for when debate helped.
The scoring function blends three criteria. Value improvement estimates immediate performance gain. Information gain estimates how much a result would clarify which direction to pursue next. Exploration utility estimates follow-up options a proposal unlocks under the remaining budget. Weights on those terms adapt with recent progress and leftover time, rather than staying fixed as pure exploitation or pure exploration.
What did JANUS score on MLS-Bench-Lite?
MLS-Bench-Lite is the authors’ headline controlled benchmark: thirty research tasks across twelve domains in which an agent must improve one targeted component of an ML system under protected trainers and multi-setting evaluation. Table 1 in the PDF compares JANUS with AutoResearch, AIDE, MLEvolve, AIBuildAI, Arbor and Codex under shared protocols.
With DeepSeek-V4 Flash as the common backbone, JANUS reports 100% valid submissions, 100% Beat Any, 76.7% Beat Median, 43.3% Beat All, a 41.99 development score and a 41.07 test score. Codex on the same backbone sits at 38.91 test. Stronger backbones raise the ceiling: JANUS with GPT-5.6 Sol reaches 46.88 test, and wrapping the JANUS loop around Codex with GPT-5.6 Sol reaches 49.78.
Those are author-run numbers on their hardware (single A100, five-hour wall-clock budget per MLS task). Beat All means the final submission beat every provided human baseline on a task; it is not an independent human study of JANUS itself.
| Method | Valid Sub. | Beat All | Test Score |
|---|---|---|---|
| AIDE | 93.3% | 23.3% | 31.11 |
| Arbor | 100.0% | 33.3% | 37.34 |
| Codex | 96.7% | 36.7% | 38.91 |
| JANUS | 100.0% | 43.3% | 41.07 |
| JANUS + GPT-5.6 Sol | 100.0% | 56.7% | 46.88 |
| JANUS loop + Codex (GPT-5.6 Sol) | 100.0% | 63.3% | 49.78 |
How efficient is it on real research tasks?
Beyond the lite benchmark, the authors run three end-to-end tasks: nanoGPT training optimisation (validation bits per byte), MinGPT causal-attention kernel speedup from KernelBench, and genotype-to-phenotype prediction for maize hybrids. Figure 3 in the paper plots best-so-far curves; Table 2 converts those curves into sample-efficiency speedups against other agents.
Against Codex, JANUS’s average speedups are 7.53× on nanoGPT, 2.19× on MinGPT and 1.51× on G2P, for a 3.74× mean. Peak speedups reach 43× on nanoGPT. The metric is not wall-clock alone: it asks how many experiments each method needed to first hit the same performance threshold on a shared reachable interval.
The authors also argue JANUS keeps improving after plateaus where baselines stall, and that gains are not just copying strong human baselines: similarity-weighted baseline analysis in the paper shows high JANUS scores even when code resemblance to top baselines is low.
What do the ablations show?
Ablations matter because AutoResearch papers often stack many moving parts. Removing adaptive priority weighting in favour of exploitation-only or exploration-only hurts all three real tasks. Disabling world-model-guided refinement of proposals also slows progress when search stagnates.
Dropping the world model entirely, or freezing evolution for either the research agent or the world model, likewise lowers final performance. The paper’s reading is that the two evolution processes are complementary: without world-model evolution, runs stall on plateaus; without research-agent evolution, improvement is slower throughout.
A dynamic-strategy analysis on nanoGPT, scored by an LLM judge on proposal text, shows exploration emphasised early, exploitation rising as winners appear, and information-gain spikes preceding breakthroughs—an intuitive story, though the judge itself is another model.
How does this compare with other AutoResearch systems?
JANUS lands in a busy AutoResearch shelf that already includes Karpathy-style single-GPU loops, tree-search scientist agents, and Microsoft’s Fork-and-Flush basin analysis. The distinctive claim here is not another scaffold of roles, but a learned, co-evolving predictor that decides which branch is worth the next GPU hour.
Benchmarks that ask coding agents to build agents, such as AgentSWE, and harnesses that stress long-context evidence banks like LongHarness, measure adjacent skills. JANUS’s contribution is budget-aware scientific search rather than a new coding leaderboard.
World-model language for agents also shows up in robotics and video systems such as Aether’s CRIS-0. JANUS applies a similar forecasting intuition to research graphs instead of physical trajectories.
What should readers watch next?
Readers should treat Table 1 and Table 2 as author-reported until independent groups rerun MLS-Bench-Lite and the three research tasks with the public code. Check whether the GitHub repository ships the full harness adapters the paper describes, and whether DeepSeek-V4 Flash and GPT-5.6 Sol endpoints used in the study remain available under the same configs.
Watch follow-up work that swaps the world-model prompt stack for a smaller trained value head, or that stress-tests JANUS when the executor’s failure modes differ from the predictor’s training distribution. Those are the usual places co-evolving loops overfit.
For teams already running AutoResearch overnight, the practical experiment is narrow: keep your executor, add a proposal-scoring debate step before launch, and measure experiments-to-threshold rather than only final score.
Common questions
Is JANUS a new foundation model?
No. It is a research-agent framework that wraps existing backbones (the paper emphasises DeepSeek-V4 Flash and GPT-5.6 Sol) with a Predict–Verify–Evolve loop and a co-evolving world model.
What does a 3.74× speedup mean here?
On the authors’ definition, it is the average ratio of how many experiments a baseline needed versus JANUS to reach the same performance thresholds on nanoGPT, MinGPT and genotype-to-phenotype tasks. It is not a claim about token throughput.
Where is the code?
The paper lists https://github.com/bilemond/JANUS as the project page. Availability and license of individual harness pieces should be checked on that repository before you depend on a reproduction.
What to remember
JANUS argues that AutoResearch should forecast before it spends, then turn every run into skills for both the agent and the predictor. The MLS-Bench-Lite and speedup tables are strong on the authors’ terms; independent reproduction is the next evidence to demand.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards
