Look Before You Launch: Learned Throughput Prediction for Hardware Selection in a Production Training Fleet

Hossein Askari, Wei Sun, Chenguang Zhu, Bin Zan, Sreen Tallam, Keyu ManarXiv 2026 · arXiv

TL;DR

Meta researchers introduced LEAP, a machine learning service that predicts the throughput of AI training jobs across diverse hardware types before they run. By replacing rigid lookup tables with a binned cascade of gradient-boosted trees, LEAP achieved 77% prediction accuracy and reduced overall computational costs by approximately 3% across tens of thousands of daily job attempts.

What Problem Does This Paper Address?

In massive industrial compute fleets, "hardware" is no longer a monolith. Modern training environments mix accelerators from different vendors and generations. While a training job might run successfully on several different GPU types, its performance—measured in samples per second or Queries Per Second (QPS)—can vary dramatically.

Choosing the wrong hardware doesn't cause a crash; it causes an "efficiency leak." If a job runs 20% slower than it could on a similarly priced alternative, that's wasted money and energy. The challenge is that fleet schedulers need an accurate throughput estimate at submission time to make the best routing decision.

Previous solutions relied on simple lookup tables that suffered from two major issues:

  1. Low Coverage: They only worked for a handful of common model types, leaving the "long tail" of diverse workloads unserved (see Figure 1).
  2. Rigidity: They couldn't account for subtle changes in job configuration or the inherent noise in large-scale distributed systems.
Figure 1: Routing must serve a long tail of model types. Cumulative share of one day’s job attempts over model types ranked by volume.

Methods

The authors developed LEAP (Learned Throughput Prediction), a system integrated directly into the job launcher.

The Architecture

As shown in Figure 2, LEAP operates in a continuous loop. Every night, it retrains on the previous 30 days of fleet data. When a new job is submitted, the system extracts features, predicts throughput for all candidate hardware, and generates a "preference list" for the cluster scheduler.

Figure 2: LEAP architecture showing the nightly retraining loop and the real-time inference path.

The "Binned Cascade" Model

Predicting throughput is difficult because values span seven orders of magnitude—from single digits to over a million QPS. A standard regression model tends to ignore the "extremes" to minimize average error.

To solve this, LEAP uses a binned cascade of Gradient-Boosted Decision Trees (GBDTs):

  1. Binning: It divides the throughput range into 13 equal-frequency bins.
  2. Classification: A GBDT classifier first predicts which bin a job belongs to.
  3. Expert Regression: Each bin has its own "expert" GBDT regressor trained specifically for that performance range.
  4. Inference: The system picks the most likely bin and uses that bin's expert to provide the final log-throughput prediction (see Figure 6).
Figure 6: The prediction model uses a bin classifier to route jobs to one of 13 specialized expert regressors.

Datasets

The study uses internal production data from Meta’s training fleet.

  • Scale: Approximately 25,000 job attempts per day.
  • Diversity: Over 660 production model types and more than 20 distinct hardware types.
  • Features: 56 numeric and 17 categorical features, including job configuration (batch size, trainer count), hardware specs (memory bandwidth, peak TFLOPS), and job history.

Key Insights and Ideas

  • History is the Best Predictor: The single most important feature for accuracy is the job's own prior performance. As seen in Figure 4, "prior log throughput" accounts for the largest share of permutation importance.
  • GBDTs Beat LLMs for Tabular Data: The researchers tested fine-tuned Large Language Models (like Llama 4 and Gemma 3) and Graph Neural Networks (GNNs). Surprisingly, the GBDTs were more accurate and significantly cheaper to run. GNNs matched GBDT accuracy but required expensive GPU inference and complex graph caching that wasn't justifiable for a 3% cost saving.
  • Log-Space is Essential: Because of the massive dynamic range of throughput, training in log-space (predicting ln(QPS)) was critical for stable convergence and meaningful error metrics.
Figure 4: Permutation importance shows that job history and scale are the primary drivers of prediction accuracy.

Key Results

  • Accuracy: LEAP achieves a median w20 accuracy of 77% (meaning 77% of predictions are within ±20% of the actual measured value).
  • Coverage: While the legacy lookup table only provided estimates for 13% of jobs, LEAP provides an estimate for 100% of eligible jobs.
  • Economic Impact: By enabling better hardware routing, LEAP reduced the total computational infrastructure cost of the fleet by roughly 3%. Over six months, the share of GPU-hours receiving a LEAP ranking grew from 58% to 95%.

Limitations and Future Work

  • Cold Starts: LEAP struggles with "novelty." It cannot reliably rank a brand-new hardware type or a completely new model architecture until it has seen at least a few real-world runs to populate its history.
  • Label Noise: Throughput in a production fleet is noisy due to network congestion and multi-tenancy. There is likely a "noise ceiling" that no model can overcome without more granular telemetry.
  • Future Hybridization: The authors suggest a future version could use a GNN specifically to "bootstrap" predictions for novel model architectures that haven't established a history yet, while keeping the efficient GBDT cascade for the bulk of the fleet.

Summary updated

Abstract

In mixed-vendor accelerator fleets, many training jobs can run on several hardware types, and the best choice depends on each job's throughput on each type. Routing a job needs a throughput estimate for every candidate type before the job runs. We present LEAP, a machine learning performance prediction service that predicts each training job's throughput on every candidate hardware type at submission time and ranks the candidates by predicted throughput per unit cost. It routes tens of thousands of training job attempts a day across hundreds of different model types and more than 20 hardware types in a production fleet. LEAP predicts throughput from submission-time training configuration, including each job's prior throughput. LEAP reaches an accuracy of 77% over 30 nightly retrains, higher than the lookup table it replaced and at least as high as every alternative model we tested, including fine-tuned language models. Over six months, the share of eligible training GPU-hours that received a ranking by LEAP grew from 58% to 95%, and we estimate that it saves about 3% of training computational cost. In this paper, we identify the drivers that actually move accuracy and those that do not. We offer these as lessons for teams building performance predictors at fleet scale.
BibTeX
@misc{askari2026look,
  author = {Hossein Askari and Wei Sun and Chenguang Zhu and Bin Zan and Sreen Tallam and Keyu Man},
  title = {{Look Before You Launch: Learned Throughput Prediction for Hardware Selection in a Production Training Fleet}},
  year = {2026},
  eprint = {2611.04057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04057},
}
Figure