How We Fixed GPU Cluster Scheduling with Budgets, Fair-Share, and Time-Slicing

The GPU Scheduling Problem Nobody Talks About

You have 1,000 GPUs. You have 150 researchers. Demand is 2-3x capacity. Your priority-based scheduler just produced a world where 100% of jobs are marked HIGH priority, users park idle workloads to "reserve" GPUs, and your on-call engineers spend most of their time negotiating shutdowns instead of fixing things.

This is the classic tragedy of the commons playing out in AI infrastructure. Ai2's infrastructure team recently published a detailed account of how they replaced their priority scheduler with a system built on GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract. The results are worth studying for anyone running shared compute.

The Four-Layer Metrics Pyramid

Ai2 frames cluster health as a pyramid:

1. Availability – Is the hardware healthy?
2. Occupancy – Is available time assigned to a workload?
3. Impact – Are the most valuable workloads getting resources?
4. Utilization – Is the GPU actually busy during a workload's lifetime?

Most teams optimize availability and occupancy. The hard part is impact—and that's where scheduling policy matters. For a complementary look at how agentic workloads are changing infrastructure demands, see our analysis of on-device function calling with Google AI Edge Gallery.

The Solution: Budgets, Fair-Share, and a Contract

Step 1: Replace GPUs with GPU Time Budgets

Instead of assigning teams monopolies over GPU sets (which caused idle capacity due to research seasonality), Ai2 allocates portions of GPU time. Managers act like investors: before workloads exist, they decide how to fund each research effort based on expected impact.

Key insight: every request for GPU time must be funded by a budget, or it is not protected from preemption. Nothing is free. Squatting now costs your own team's budget.

Step 2: Hierarchical Fair-Share Scheduler

The scheduler tracks occupancy over a sliding lookback window (default: 7 days) and sorts workloads from under-utilized allocations above over-utilized ones. The algorithm isn't new—it traces back to Hadoop Fair Scheduler (2009) and is used in SLURM's Fair Tree and YARN. What's new is the input: the tree mirrors the research program structure, and weights are manager-set budgets rather than static quotas.

# Simplified fair-share sorting logic (conceptual)
# Each allocation has a weight (budget) and actual usage over lookback window

allocations = {
    'project_a1': {'budget': 0.35, 'usage': 0.28},  # under-utilized
    'project_b2': {'budget': 0.20, 'usage': 0.25},  # over-utilized
    'project_c3': {'budget': 0.45, 'usage': 0.47},  # over-utilized
}

# Sort workloads: under-utilized allocations get priority
def fair_share_key(alloc):
    # Lower ratio = higher priority
    return alloc['usage'] / alloc['budget']

sorted_allocs = sorted(allocations.items(), key=lambda x: fair_share_key(x[1]))
# Result: project_a1 (0.8) > project_b2 (1.25) > project_c3 (1.04)
# Wait, c3 (1.04) should be before b2 (1.25)
# Corrected: project_a1 (0.8) > project_c3 (1.04) > project_b2 (1.25)
print(sorted_allocs)

Step 3: The Scheduling Contract

Each workload declares a minimum runtime—the shortest occupancy needed to make meaningful progress. During this window, it's protected from preemption. After that, the scheduler can rebalance, automatically re-queueing resumable workloads. Users can set minimum runtime to zero for fully preemptible, unallocated work.

The workload lifecycle:

1. Submit with minimum runtime and resumability flag.
2. Scheduled by fair-share, weighted by occupancy/budget ratio.
3. Runs for minimum runtime (charged to budget).
4. May continue if allocation still prioritizes it (also charged).
5. May be preempted and requeued (back to step 2).
6. Completes, releasing resources.

The Results

• 98% of owed GPU hours delivered over 30 days; 13 of 15 teams got ≥95%.
• Occupancy held at 98% before and after, with demand still 2-3x capacity.
• Debug workload p90 wait: 2 hours → 30 seconds.
• Median queue wait on largest H100 cluster: 5 min → 24 sec.
• On-call repairs requiring human intervention: down 74%.
• 18% of delivered time was unallocated—absorbing demand mismatches.

For a related trend in how serverless knowledge and MCP are reshaping agent infrastructure, check our Foundry IQ serverless preview breakdown.

Limitations and Pitfalls

What Didn't Work

• Interactive sessions suffered. Researchers used to hold dev sessions for up to a week. Time-slicing imposed an 8-hour cap on protected runtime, after which sessions became preemptible. Losing volatile state was painful. Ai2 is now building a CPU-only cluster for dev sessions and restorable sessions.
• The learning curve was steeper than expected. Incremental rollout meant different clusters behaved differently. Old terminology (like "priority") persisted in interfaces with changed meanings. Documentation alone didn't fix confusion—live explanatory sessions did.
• Capacity fragmentation is a suspected emerging issue. Minimum runtime protection may reduce the scheduler's ability to interrupt many jobs at once to place a large pending workload, potentially increasing wait times for the largest jobs. Ai2 is using simulations to investigate.

Critical Considerations

• Budget review is a political process. Moving from case-by-case negotiation to administrative budgeting is an improvement, but it centralizes power in managers. The system assumes managers have the most context on tradeoffs—which may not always hold.
• Simulation accuracy matters. Ai2's predictions were directionally correct but off by an order of magnitude in some cases (predicted 6h→5min for debug waits; actual 2h→30s). Simulations are necessary but not sufficient.
• The 7-day lookback window is a tunable knob. Too short and bursty teams get starved; too long and the system responds slowly to changing priorities.

Next Steps for Learners

• Study the original Dominant Resource Fairness paper (Ghodsi et al., 2011) for the theoretical foundation.
• Explore SLURM's Fair Tree and YARN's Fair Scheduler implementations for production-grade reference.
• Build a small simulator yourself: model workloads with (gpus, runtime, minimum_runtime, budget) and implement a fair-share queue. You'll learn more from 200 lines of simulation than from reading ten papers.
• If you're running shared GPU infrastructure, audit your current scheduler for the three pathologies Ai2 named: squatting, priority inflation, and on-call toil from non-preemptable workloads.

The Bottom Line

Ai2's core insight is simple but powerful: stop trying to solve the scheduling puzzle directly; make leadership think like investors instead. Decide how to fund research efforts with GPU time before workloads exist, then let a fair-share scheduler enforce those budgets with a time-slicing contract.

The results—98% allocation delivery, 98% occupancy, debug waits down from hours to seconds, 74% less on-call toil—show this isn't just theory. It's a production system serving 150 researchers on thousands of H100/B200/B300 GPUs.

If you're building or operating shared compute for AI workloads, this is required reading. The full write-up is available as 근거자료 on Hugging Face's blog.

Related reading:

• On-Device Function Calling: Google AI Edge Gallery Brings Agentic AI to Mobile
• Microsoft Foundry IQ: Build Smarter Agents Faster with Serverless Knowledge and MCP