Automated Machine Learning via ADMM

Sijia Liu, Parikshit Ram, Djallel Bouneffouf, Gregory Bramble, Andrew R. Conn, Horst Samulowitz, Alexander G. Gray

Abstract

We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints along-side the black-box optimization objective. We empirically evaluate the flexibility, effectiveness, and unique capability of our proposed scheme on a collection of binary classification data sets from UCI ML & OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.

Introduction

Building an effective machine learning pipeline is a complex task that involves selecting the right algorithms for preprocessing, feature selection, and modeling, while simultaneously fine-tuning their hyperparameters. This challenge, known as the Combined Algorithm Selection and Hyperparameter optimization (CASH) problem, is characterized by its mixed-variable nature (categorical choices vs. continuous values) and its black-box objective function, which lacks gradient information and is expensive to evaluate.

Existing AutoML tools like Auto-sklearn and TPOT often struggle with the tight coupling between algorithm selection and hyperparameter optimization (HPO). Furthermore, they typically lack the ability to incorporate real-world black-box constraints—such as prediction latency, memory footprint, or fairness—directly into the search process. This paper introduces a novel framework that leverages the Alternating Direction Method of Multipliers (ADMM) to address these limitations by decomposing the CASH problem into manageable sub-problems.

Methodology: The ADMM-Based Framework

The core innovation of this work is the use of ADMM to decouple the algorithm selection and HPO tasks. While ADMM is traditionally used for large-scale convex optimization, the authors adapt it for the non-convex, black-box AutoML domain. The framework decomposes the global optimization problem into three iterative sub-problems:

  1. Hyperparameter Optimization (the $\theta$-min problem): This sub-problem optimizes the continuous hyperparameters for a fixed set of algorithms. It is solved using standard Bayesian Optimization (BO).
  2. Integer Projection (the $\delta$-min problem): This step maps continuous surrogate variables back to valid discrete algorithm settings, bridging the gap between continuous optimization and categorical choices.
  3. Algorithm Selection (the $z$-min problem): This sub-problem selects the best algorithms for each pipeline stage. The authors model this as a Combinatorial Multi-Armed Bandit (CMAB) problem and employ Thompson Sampling for efficient exploration.

By introducing continuous relaxations for discrete choices, the framework allows ADMM to use "soft" assignments during optimization before projecting them to "hard" choices. Crucially, black-box constraints are integrated into the ADMM iterations via auxiliary variables and penalty terms, steering the search toward high-performing, deployable models.

Experimental Evaluation

Datasets and Setup

The framework was evaluated on 30 binary classification datasets from the UCI Machine Learning Repository, OpenML, and Kaggle. Additionally, the Home Credit Default Risk dataset from Kaggle was used to test the framework's ability to handle complex constraints like prediction latency and fairness.

Performance Benchmarking

In a head-to-head comparison across the 30 datasets, the ADMM-based framework outperformed existing state-of-the-art tools. It achieved the best performance on 50% of the datasets, significantly higher than Auto-sklearn (27%) and TPOT (20%).

Mean rank of ADMM compared to baselines across 30 datasets.

As shown in the figure above, the ADMM framework (teal) consistently achieves a superior mean rank across datasets compared to Auto-sklearn (ASKL), TPOT, and Random Search (RND). Notably, the decomposition approach allows the framework to reach optimal performance levels up to 10x faster than joint optimization methods.

Constraint-Aware Optimization

A unique capability of the ADMM framework is its explicit handling of black-box constraints. On the Home Credit dataset, the framework successfully maximized AUROC while strictly adhering to user-defined limits for prediction latency and fairness (measured by disparate impact thresholds).

Performance of the ADMM framework under varying latency and fairness constraints.

The results demonstrate that the constrained version of the framework (CST) effectively maintains high performance while satisfying stringent thresholds, whereas unconstrained methods (UCST) often produce models that are undeployable due to excessive latency or unfairness.

Conclusion

The proposed ADMM-based framework represents a significant step forward in AutoML by effectively decomposing the complex CASH problem and incorporating black-box constraints. Its modular design allows for the integration of various solvers, making it a flexible meta-framework for pipeline optimization. While empirical results are highly promising, showing substantial gains in both accuracy and speed, future work will focus on establishing theoretical convergence guarantees for this non-convex application and exploring multi-fidelity optimization to further reduce computational costs.

Summary updated

Abstract

We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints along-side the black-box optimization objective. We empirically evaluate the flexibility (in utilizing existing AutoML techniques), effectiveness (against open source AutoML toolkits),and unique capability (of executing AutoML with practically motivated black-box constraints) of our proposed scheme on a collection of binary classification data sets from UCI ML& OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.
BibTeX
@inproceedings{liu2020automated,
  author = {Sijia Liu and Parikshit Ram and Djallel Bouneffouf and Gregory Bramble and Andrew R. Conn and Horst Samulowitz and Alexander G. Gray},
  title = {{Automated Machine Learning via ADMM}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5926},
}
Figure