CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performance

Wei Wen, Quanyu Zhu, Weiwei Chu, Wen-Yen Chen, Jiyan YangarXiv 2024 · OpenReview

Abstract

Scaling up deep learning models has been proven effective to improve the intelligence of machine learning (ML) models, especially for industry recommendation models and large language models. The co-design of large distributed ML systems and algorithms to maximize training performance plays a pivotal role in this success. As models scale, the number of co-design hyper-parameters grows rapidly, making it challenging to find optimal setups for system performance. We propose CubicML, which uses an ML model as a proxy to predict training performance for search efficiency and flexibility. We demonstrate that CubicML effectively optimizes the training speed of in-house ads recommendation models with 73 billion parameters and large language models up to 405 billion parameters at Meta.

Introduction

As deep learning models scale to hundreds of billions of parameters, the infrastructure required to train them becomes incredibly complex. Engineers must "co-design" the system and the algorithm, choosing between various parallelism strategies (Data, Tensor, Pipeline), sharding techniques (FSDP/ZeRO), and precision formats (FP8, BF16).

The search space for these hyper-parameters is astronomical—often in the millions of possible combinations. Traditionally, human experts manually tune these settings, but this process is slow, expensive, and must be repeated every time the model architecture or hardware changes. CubicML aims to replace this manual "guess-and-check" process with an automated, data-driven approach.

The CubicML Framework

CubicML operates as a closed-loop AutoML framework consisting of five key components:

  1. ML Systems: The GPU clusters where training jobs are executed.
  2. Historical Job Data: A database storing metadata (configurations and resulting speeds) from completed jobs.
  3. Search Space: A defined range of tunable parameters, such as batch size and per-layer sharding strategies.
  4. The Predictor: A lightweight regression model (e.g., neural network or gradient-boosting tree). Crucially, it is trained using Margin Ranking Loss, meaning it prioritizes correctly ranking configurations by speed rather than predicting exact absolute values.
  5. The Searcher: An optimization algorithm (focused on Reinforcement Learning via the REINFORCE algorithm) that queries the Predictor to find the most promising configurations to launch.
The CubicML framework architecture showing the closed-loop optimization process between the predictor, searcher, and ML systems.

The process is iterative: the system samples configurations, runs them, saves the performance data, and retrains the predictor to improve its accuracy over time.

Experimental Evaluation

The paper evaluates CubicML on two massive, real-world production workloads at Meta:

  • Ads Recommendation Models: A "Wukong" backbone model with 73 billion parameters (excluding embeddings), trained on Meta's "Grand Teton" platform with 128 NVIDIA H100 GPUs.
  • Large Language Models (LLMs): A suite of models ranging from 27 million to 405 billion parameters (Llama 3 scale). The dataset included 568 LLM training jobs accumulated over time to test the predictor's accuracy across different scales and hardware setups.

Key Results and Insights

Performance Optimization

In the ads recommendation use case, CubicML found a configuration that was 10.3% faster in terms of Queries Per Second (QPS) than the baseline established by expert human engineers. In industrial-scale AI, a 10% efficiency gain translates to significant savings in electricity and hardware costs.

Performance results: (a) QPS improvement over samples compared to human and random baselines, and (b) correlation between predicted and ground-truth QPS.

Predictor Accuracy and Efficiency

  • Ranking over Regression: The authors found that the predictor doesn't need perfect absolute accuracy. As long as it maintains a high rank correlation (knowing Config A is faster than Config B), the searcher can successfully find the global optimum.
  • Data Efficiency: A high rank correlation (Pearson/Spearman > 0.9) is reached after only 150 training examples, meaning thousands of trial runs are not required to build a robust performance model.
  • LLM Accuracy: For LLMs, the predictor achieved a Kendall Tau rank correlation of 0.88 and a Pearson correlation of 0.97, proving its capability to model the performance of the world's largest AI models.

Discussion and Future Work

While CubicML demonstrates strong performance, it faces challenges with distribution shift. The model performs best with "fresh" data; performance correlation drops when predicting future jobs based on data from months prior due to updates in system software or hardware wear. Additionally, while the model can generalize from small-scale trials (e.g., 8 GPUs) to larger scales (up to 16,384 GPUs), out-of-distribution scaling remains more difficult than in-distribution prediction.

Future work will focus on more robust online learning techniques to adapt to hardware and software evolution fluidly without losing historical knowledge, further reducing the need for initial random search phases.

Summary updated

Abstract

Scaling up deep learning models has been proven effective to improve intelligence of machine learning (ML) models, especially for industry recommendation models and large language models. The co-design of large distributed ML systems and algorithms (to maximize training performance) plays a pivotal role for its success. As it scales, the number of co-design hyper-parameters grows rapidly which brings challenges to feasibly find the optimal setup for system performance maximization. In this paper, we propose CubicML which uses ML to automatically optimize training performance of large distributed ML systems. In CubicML, we use an ML model as a proxy to predict the training performance for search efficiency and performance modeling flexibility. We proved that CubicML can effectively optimize training speed of in-house ads recommendation models with 73 billion parameters and large language models up to 405 billion parameters at Meta.
BibTeX
@misc{wen2024cubicml,
  author = {Wei Wen and Quanyu Zhu and Weiwei Chu and Wen-Yen Chen and Jiyan Yang},
  title = {{CubicML: Automated ML for Large ML Systems Co-design with ML Prediction of Performance}},
  year = {2024},
  eprint = {2409.04585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.04585},
}
Figure