Abstract
We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints along-side the black-box optimization objective. We empirically evaluate the flexibility, effectiveness, and unique capability of our proposed scheme on a collection of binary classification data sets from UCI ML & OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.
Introduction
The configuration of machine learning pipelines—selecting a sequence of modules like preprocessors and estimators and tuning their hyperparameters—is known as the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem. This task is inherently difficult due to the presence of mixed variables (discrete algorithm choices vs. continuous hyperparameters), the black-box nature of the objective function (no gradients), and the tight coupling between algorithm choice and hyperparameter effectiveness. Furthermore, real-world applications often impose black-box constraints, such as latency or fairness, which standard AutoML tools typically struggle to incorporate directly into the search process.
The ADMM-Based Framework
To address these challenges, we propose a framework based on the Alternating Direction Method of Multipliers (ADMM). Traditionally used for convex optimization, ADMM is creatively adapted here to decouple the complex CASH problem into three iterative, more manageable sub-problems:
- Hyperparameter Optimization ($\theta$-min): With algorithm choices fixed, the framework optimizes continuous hyperparameters using Bayesian Optimization (BO).
- Integer Projection ($\delta$-min): This step projects continuous surrogate variables back into the discrete integer set required for algorithm selection.
- Algorithm Selection ($z$-min): Algorithm combinations are treated as a Combinatorial Multi-Armed Bandit (CMAB) problem, solved using Thompson Sampling to explore the most promising pipeline structures.
Incorporating Black-Box Constraints
A key innovation of this framework is its ability to handle black-box constraints (e.g., latency or fairness). By introducing auxiliary variables and Lagrangian multipliers, constraints are integrated directly into the optimization objective. This steers the search toward feasible regions of the pipeline space, rather than treating constraints as post-hoc filters.
Experimental Evaluation
The framework was evaluated against state-of-the-art AutoML tools, Auto-sklearn (ASKL) and TPOT, across 30 binary classification datasets from the UCI ML and OpenML repositories.
Performance and Efficiency
The ADMM-based approach demonstrated superior performance, achieving the best results on 50% of the tested datasets. In terms of efficiency, the decomposition strategy provided up to a 10x speedup compared to joint optimization methods.
Constraint Satisfaction
On the Kaggle Home Credit Default Risk dataset, the constrained version of the framework (CST) significantly outperformed unconstrained versions (UCST) that relied on post-hoc filtering. As latency and fairness constraints became more stringent, the ADMM framework's ability to incorporate these constraints into the search guidance proved critical for finding high-performing, valid pipelines.
Conclusion
The proposed ADMM framework offers a modular and effective solution to the CASH problem. By decomposing the optimization and integrating black-box constraints, it achieves significant performance gains and practical utility over existing tools. Future work will explore theoretical convergence in non-convex settings and extending the framework to more complex, arbitrary graph-based pipeline architectures.
Summary updated