Abstract: AutoML has become a popular service that is provided by most leading cloud service providers today. In this paper, we focus on the AutoML problem from the service provider’s perspective, motivating the need for multi-device, multi-tenant model selection. We introduce MM-GP-EI, an algorithm designed to coordinate multiple computation devices across multiple users simultaneously, minimizing total regret with near-linear speedups.
Introduction
Most Automated Machine Learning (AutoML) research assumes a "single-tenant" scenario: one user trying to find the best model for one dataset using their own hardware. In reality, major cloud providers serve thousands of users at once, all sharing a finite pool of GPUs and CPUs. This paper shifts the focus to the cloud service provider, introducing MM-GP-EI, the first algorithm designed to maximize "global happiness" by coordinating multiple computation devices across multiple users.
The Multi-Tenant Scheduling Challenge
The service provider faces a complex scheduling challenge: which user should get the next available GPU, and which model configuration should they test? If the provider simply lets users run their own independent searches, hardware is often used inefficiently. Conversely, if the provider tries to optimize for everyone at once, they need a mathematically sound way to balance "exploration" (trying new models) and "exploitation" (refining known good models) across a diverse population of users.
The goal is to maximize "global happiness"—a metric representing the collective progress of all users toward finding their respective optimal models.
Methodology: MM-GP-EI
The authors build upon GP-EI (Gaussian Process Expected Improvement), a gold-standard algorithm for Bayesian Optimization. Their novel contribution is the Multi-device, Multi-tenant GP-EI (MM-GP-EI).
Key components include:
- EIrate (Expected Improvement Rate): Instead of just looking at how much a model might improve a user's results, the algorithm considers the "bang for the buck"—the expected improvement divided by the time (cost) it takes to run that model.
- Global Acquisition: When a device becomes free, the system calculates the total expected improvement across all users who could benefit from a specific model. It then selects the model that offers the highest collective "EIrate."
- Maximum Incremental Uncertainty (MIU): To prove their algorithm works theoretically, the authors introduced MIU. This measures how much "new information" a model provides, working even when there is zero observation noise.
Experimental Evaluation
The researchers evaluated their approach using three benchmarks:
- DeepLearning Dataset: 22 users choosing from 8 neural network architectures (e.g., AlexNet, ResNet-50) for image classification.
- Azure Dataset: 17 users choosing from 8 binary classifiers available on Microsoft Azure.
- Synthetic Dataset: 50 users and 50 models, used to test scaling as devices increase to 8.
Key Results
The MM-GP-EI algorithm demonstrates significant performance gains over standard baselines.
- 5x Faster Convergence: On the Azure dataset, MM-GP-EI reached the same level of global performance up to 5 times faster than standard "Round Robin" or random scheduling strategies.
- Linear Scaling: The algorithm achieves near-linear speedup. Doubling the number of GPUs roughly halves the time it takes for the community of users to reach a certain level of "happiness."
- Theoretical Guarantees: The authors proved a regret bound of $O((MIU(T,K) + M) \frac{N^2}{M} \bar{c})$, where $N$ is users and $M$ is devices, showing that regret decreases significantly as more devices are added.
Conclusion
MM-GP-EI provides a robust framework for multi-tenant AutoML. While it currently assumes known runtimes and homogeneous devices, it sets the stage for more efficient cloud-scale machine learning services. Future work will explore heterogeneous resources and continuous hyperparameter tuning spaces.
Summary updated