How Ai2 Fixed GPU Scheduling Across Thousands of H100 and B200s

👁 112 reads

Allen Institute for AI (Ai2) published a detailed breakdown of how it rebuilt job scheduling across its GPU clusters - offering a textbook case study in how well-designed infrastructure buckles under real-world research pressure.

Ai2 manages thousands of NVIDIA H100, B200, and B300 GPUs organized into clusters ranging from 88 to 1,024 GPUs. Around 150 internal researchers rely on this hardware for vastly different workloads: training large language models (LLMs) and vision-language models (VLMs), reinforcement learning simulations for robotics, and post-training for scientific agent workflows. That workload profile demands both long, uninterrupted training sessions and low-latency access for interactive debugging - two requirements that are notoriously hard to balance with a single priority scheme.

A classic tragedy of the commons

The core issue was neither new nor subtle: demand for GPU time outstripped supply by a wide margin. At any given moment, queued jobs requested 2 to 3 times more cards than were physically available. In other words, every available compute hour had two to three distinct research projects competing for the same hardware.

The legacy setup relied on priorities and allowed jobs to opt out of preemption - the ability to pause or terminate a workload to make room for a higher-priority task. Each team had an allocation cap of non-preemptible GPUs, while preemptible workloads could opportunistically grab idle capacity above that threshold. On paper, that setup looks clean. In practice, predictable bottlenecks emerged.

First came GPU squatting: researchers ran empty “no-op” jobs simply to hold hardware reservations they could jump into when debugging code, because launching a fresh interactive job with low latency was nearly impossible. Second came priority inflation: eventually, 100% of scheduled jobs ran at the highest “HIGH” priority tier, starving lower priorities entirely. Third - and perhaps most taxing on daily operations - on-call engineers spent the majority of their shift manually negotiating the shutdown of non-preemptible jobs running on nodes flagged for hardware failures.

Ai2 admits it took time to recognize the root cause. Initial workarounds focused on stricter priority enforcement, followed by bypassing the scheduler altogether to grant specific projects dedicated cluster monopolies. Only later did the team identify this as a textbook tragedy of the commons: an environment where individual actors, operating rationally in their own self-interest, collectively degrade a shared resource in a way nobody intended.

The fix involved replacing the priority-based scheduler with a three-part model: upfront GPU time budgets assigned to projects, hierarchical fair-share allocation, and explicit time-slicing contracts defining clear windows during which a project is guaranteed resources. Crucially, the shift was organizational rather than purely algorithmic: allocating compute moved away from ad-hoc operational friction into an administrative budgeting process.

That distinction matters for anyone operating shared compute infrastructure. Ai2 did not face a scheduler algorithm flaw; it faced misaligned incentives. When you let engineers hold GPUs without an immediate cost, they hold them. When priority levels are free, everyone picks maximum. By giving those decisions measurable budget accountability across the entire organization, behavior changed far more effectively than any preemption rule could achieve.

// read moreNVIDIA AI Agents Deployed Across European CitiesNewsStrata: 125B Model on 12 GB GPU at 94 Tokens/SecondAI ToolsJensen Huang: The Man Who Bet on AI and Won - Denny’s to $3TMagazine

Kinetyra