Ai2 replaced priority GPU scheduling with time budgets. 98% occupancy is the institute’s

What did Ai2 publish on 9 October?

The dated object is Ai2’s post “Impactful scheduling for GPU clusters,” 9 October 2026, written as “we” from the AI Infrastructure team. Hugging Face’s same-day republish under allenai/impactful-scheduling prints dateCreated 2026-10-09T15:20:29.312Z and lists Kyle Wiggers / Ai2Comms in that page’s author chrome. The official page we opened has no named byline.

The scheduler change the post describes is already in production. Ai2 says it began a cluster-by-cluster rollout at the end of July. The public artifact is the write-up, not a downloadable scheduler, a Hub repo, or a license file.

This is an on-prem research-cluster operations note, not a hosted inference product. For how operators usually split control of serving stacks, see our managed versus self-hosted guide. For the hardware metrics that sit under any occupancy claim, see AI chip metrics.

What does Ai2 say the old scheduler did wrong?

Ai2 says it manages thousands of NVIDIA H100, B200, and B300 GPUs in clusters of 88 to 1,024 GPUs for about 150 internal researchers. The work listed is LLM and VLM training, robotics reinforcement-learning simulation, and post-training for scientific agents — the same scientific-writing lane as AstaBrief 8B. Submitted demand, the post says, is two to three times available GPUs at any moment.

The old system was priority-based. Teams had a concurrent-GPU cap for jobs that opted out of preemption; preemptible jobs could overflow onto idle GPUs. Ai2 says that produced three “predictable pathologies”: GPU squatting (no-op jobs parked so a debugger could attach), priority inflation until “100% of scheduled workloads used HIGH priority,” and on-call engineers spending most ticket time negotiating shutdowns of non-preemptable jobs on unhealthy hosts.

Tighter priority rules, then GPU monopolies for important projects, sat idle between experiments. Ai2 writes that it was “manually solving a knapsack problem.”

How do GPU time budgets and fair-share work?

The replacement allocates a share of GPU time, not a set of machines. Managers proportionally fund projects and researchers. A diagram caption says parenthetical values are “the total cluster capacity assigned to a leaf project,” and the example leaf “Project A1” is given a 35% claim. Every request for protected time must be funded by a budget. Squatting now spends that budget on nothing.

Budgets are set by a lead researcher inside a project, a PI inside a program, and a lead program manager or the CEO across programs. Ai2’s aim is to make gaming the scheduler costlier than arguing for a larger budget.

The occupancy engine is hierarchical fair-share over a sliding lookback window, defaulting to seven days. Ai2 says the algorithm is not new — it cites the Hadoop Fair Scheduler (2009), SLURM Fair Tree, and YARN Fair Scheduler — and that the new inputs are a tree that mirrors the research programs and weights set by managers. Workloads from under-used allocations sort above those from over-used ones.

Chris Clark is quoted: “The new scheduler makes it feel like we have an extra 30% compute… if we had moments when we didn't need our full slot limit, that compute was basically lost. Now… we can later burst beyond our allocation limit and still see our jobs scheduled quickly and without preemption.” That sentence is Ai2’s attributed quote, not a measurement we took.

What is the eight-hour scheduling contract?

Distributed training can hold GPUs for days. Ai2’s contract requires each workload to declare a minimum runtime — “the shortest amount of occupancy required to make meaningful progress.” During that window a funded job is protected from preemption. After it, the scheduler may rebalance and re-queue resumable jobs. Setting the minimum to zero marks the job unallocated: always preemptible and not charged. The post says the maximum allowed minimum runtime is eight hours.

Allocated occupancy is charged and protected during the minimum window. Unallocated occupancy is free, unprotected, and reclaimable — Ai2’s stated way to keep GPUs busy when funded demand is seasonal.

The operational payoff Ai2 highlights is automated drains: unhealthy hosts empty as jobs reach their minimum runtime. Repairs that needed a person in the loop fell 74%. That is an on-call toil claim, not a utilization claim. For why electricity and occupancy are different numbers, see AI data-centre energy.

Which results are simulated, and which are from the rollout?

Before rollout, Ai2 built a simulator that jumps between schedulable events given requested GPU counts and runtimes. On hand-built debug cases — small jobs with a 15-minute-or-less minimum — it predicted p90 wait falling from about 6 hours to 5 minutes. The historical log, the post says, did not contain enough of those jobs to test the hypothesis on real traces.

The live 30-day test is a different table. Occupancy stayed 98% before and after, with 2–3× demand in both periods. Teams received 98% of the GPU hours they were owed, where “owed” is the allocation capped hour by hour at actual demand. Thirteen of 15 allocations received 95% or more; the worst received 90%. Unallocated occupancy was 18% of delivered GPU time.

Queue numbers on the largest H100 cluster: median wait 5 minutes to 24 seconds; p90 about 2.8 hours to 1.8 hours. Debug p90 wait on the live system: 2 hours to 30 seconds, which beat the simulated 6 hours to 5 minutes. Ai2 notes the baseline debug sample was small and higher-variance.

What did Ai2 say still fails?

Interactive analysis sessions could previously be held for a week. Under time-slicing they become preemptible after eight protected hours if they exceed their allocation, and researchers had to rebuild volatile state by hand. Ai2 says it is investing in a CPU-only cluster next to on-prem storage for data-prep sessions, plus restorable sessions for those CPU jobs. Those are roadmap sentences.

A second open question is capacity fragmentation: minimum-runtime protection may leave fewer moments when many jobs can be interrupted at once to place a large pending training job. Ai2 says it is reproducing that in the simulator and measuring it in production.

This is an evidence review of the 9 October Ai2 page and the Hugging Face republish. We did not submit a job, inspect a budget tree, or recompute occupancy.