TL;DR
Meta researchers introduced LEAP, a machine learning service that predicts the throughput of AI training jobs across diverse hardware types before they run. By replacing rigid lookup tables with a binned cascade of gradient-boosted trees, LEAP achieved 77% prediction accuracy and reduced overall computational costs by approximately 3% across tens of thousands of daily job attempts.
What Problem Does This Paper Address?
In massive industrial compute fleets, "hardware" is no longer a monolith. Modern training environments mix accelerators from different vendors and generations. While a training job might run successfully on several different GPU types, its performance—measured in samples per second or Queries Per Second (QPS)—can vary dramatically.
Choosing the wrong hardware doesn't cause a crash; it causes an "efficiency leak." If a job runs 20% slower than it could on a similarly priced alternative, that's wasted money and energy. The challenge is that fleet schedulers need an accurate throughput estimate at submission time to make the best routing decision.
Previous solutions relied on simple lookup tables that suffered from two major issues:
- Low Coverage: They only worked for a handful of common model types, leaving the "long tail" of diverse workloads unserved (see Figure 1).
- Rigidity: They couldn't account for subtle changes in job configuration or the inherent noise in large-scale distributed systems.
Methods
The authors developed LEAP (Learned Throughput Prediction), a system integrated directly into the job launcher.
The Architecture
As shown in Figure 2, LEAP operates in a continuous loop. Every night, it retrains on the previous 30 days of fleet data. When a new job is submitted, the system extracts features, predicts throughput for all candidate hardware, and generates a "preference list" for the cluster scheduler.
The "Binned Cascade" Model
Predicting throughput is difficult because values span seven orders of magnitude—from single digits to over a million QPS. A standard regression model tends to ignore the "extremes" to minimize average error.
To solve this, LEAP uses a binned cascade of Gradient-Boosted Decision Trees (GBDTs):
- Binning: It divides the throughput range into 13 equal-frequency bins.
- Classification: A GBDT classifier first predicts which bin a job belongs to.
- Expert Regression: Each bin has its own "expert" GBDT regressor trained specifically for that performance range.
- Inference: The system picks the most likely bin and uses that bin's expert to provide the final log-throughput prediction (see Figure 6).
Datasets
The study uses internal production data from Meta’s training fleet.
- Scale: Approximately 25,000 job attempts per day.
- Diversity: Over 660 production model types and more than 20 distinct hardware types.
- Features: 56 numeric and 17 categorical features, including job configuration (batch size, trainer count), hardware specs (memory bandwidth, peak TFLOPS), and job history.
Key Insights and Ideas
- History is the Best Predictor: The single most important feature for accuracy is the job's own prior performance. As seen in Figure 4, "prior log throughput" accounts for the largest share of permutation importance.
- GBDTs Beat LLMs for Tabular Data: The researchers tested fine-tuned Large Language Models (like Llama 4 and Gemma 3) and Graph Neural Networks (GNNs). Surprisingly, the GBDTs were more accurate and significantly cheaper to run. GNNs matched GBDT accuracy but required expensive GPU inference and complex graph caching that wasn't justifiable for a 3% cost saving.
- Log-Space is Essential: Because of the massive dynamic range of throughput, training in log-space (predicting
ln(QPS)) was critical for stable convergence and meaningful error metrics.
Key Results
- Accuracy: LEAP achieves a median w20 accuracy of 77% (meaning 77% of predictions are within ±20% of the actual measured value).
- Coverage: While the legacy lookup table only provided estimates for 13% of jobs, LEAP provides an estimate for 100% of eligible jobs.
- Economic Impact: By enabling better hardware routing, LEAP reduced the total computational infrastructure cost of the fleet by roughly 3%. Over six months, the share of GPU-hours receiving a LEAP ranking grew from 58% to 95%.
Limitations and Future Work
- Cold Starts: LEAP struggles with "novelty." It cannot reliably rank a brand-new hardware type or a completely new model architecture until it has seen at least a few real-world runs to populate its history.
- Label Noise: Throughput in a production fleet is noisy due to network congestion and multi-tenancy. There is likely a "noise ceiling" that no model can overcome without more granular telemetry.
- Future Hybridization: The authors suggest a future version could use a GNN specifically to "bootstrap" predictions for novel model architectures that haven't established a history yet, while keeping the efficient GBDT cascade for the bulk of the fleet.
Summary updated