A Formal Method for AutoML via ADMM

Sijia Liu, Parikshit Ram, Djallel Bouneffouf, Deepak Vijaykeerthy, Gregory Bramble, Horst Samulowitz, Dakuo Wang, Andrew R. Conn, Alexander Gray

Abstract

We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints along-side the black-box optimization objective. We empirically evaluate the flexibility, effectiveness, and unique capability of our proposed scheme on a collection of binary classification data sets from UCI ML & OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.

Introduction

Building an effective machine learning pipeline involves solving the Combined Algorithm Selection and Hyper-parameter optimization (CASH) problem. This task is inherently difficult due to the presence of mixed variables (continuous and discrete), the black-box nature of the objective function (requiring slow training/testing cycles), and the need to satisfy real-world constraints such as latency and fairness. Traditional AutoML tools often focus solely on accuracy, neglecting these critical operational constraints.

Methodology: Decomposing the CASH Problem

The proposed framework adapts the Alternating Direction Method of Multipliers (ADMM) to the non-convex, black-box domain of AutoML. By employing a "divide and conquer" strategy, the complex pipeline configuration task is decomposed into three iterative sub-problems:

  • Continuous Hyper-parameter Optimization ($\theta$-min): Optimizes continuous parameters for selected algorithms using Bayesian Optimization (BO).
  • Integer Projection ($\delta$-min): Handles integer-based hyper-parameters via Euclidean projection (rounding to the nearest valid integer).
  • Algorithm Selection ($z$-min): Selects the optimal combination of algorithms, modeled as a Combinatorial Multi-Armed Bandit (CMAB) problem solved with Thompson Sampling.

To enhance efficiency, the authors introduce Adaptive ADMM, which uses lower-precision internal solvers in early iterations to accelerate the search process, gradually increasing precision as the search converges. This approach can lead to temporal speedups of up to 150x in certain benchmarks.

Constraint-Aware AutoML

A distinguishing feature of this ADMM-based approach is its ability to handle "black-box" constraints. By integrating auxiliary variables, constraints like inference latency or fairness metrics can be incorporated directly into the optimization loop. This allows the framework to find high-performing models that are also deployable in regulated or resource-constrained environments.

Median objective value achieved under varying constraint thresholds ($d_l$), comparing constrained (CST) and unconstrained (UCST) versions of the ADMM framework.

Experimental Evaluation

The framework was tested on 30 binary classification datasets from UCI, OpenML, and Kaggle, as well as the Home Credit Default Risk (HCDR) dataset for constraint testing.

Key Results

The ADMM framework demonstrated superior performance and efficiency across the benchmark suite:

  • Performance: ADMM achieved the best results on 50% of the datasets, outperforming Auto-sklearn (27%) and TPOT (20%).
  • Efficiency: The framework achieved over a 10x speedup and 10% improvement in objective values in many cases compared to joint optimization baselines.
  • Constraint Satisfaction: On the HCDR dataset, ADMM successfully identified pipelines that met strict fairness and latency requirements, whereas unconstrained methods typically failed to find valid solutions.
Mean rank of ADMM compared to Auto-sklearn (ASKL), TPOT, and random search (RND) over optimization time. Subplots (b) and (c) show pairwise comparisons.

Conclusion and Future Work

While the framework lacks the theoretical convergence guarantees of convex ADMM, its empirical success is notable. The decomposition approach significantly reduces the search space complexity, leading to massive temporal speedups. Future work includes extending the framework to handle flexible pipeline topologies and expanding the library of supported algorithms. The ability to treat operational constraints as first-class citizens marks a significant step toward making AutoML tools more practical for real-world applications.

Summary updated

Abstract

We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints along-side the black-box optimization objective. We empirically evaluate the flexibility (in utilizing existing AutoML techniques), effectiveness (against open source AutoML toolkits),and unique capability (of executing AutoML with practically motivated black-box constraints) of our proposed scheme on a collection of binary classification data sets from UCI ML& OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.
BibTeX
@inproceedings{liu2020formal,
  author = {Sijia Liu and Parikshit Ram and Djallel Bouneffouf and Deepak Vijaykeerthy and Gregory Bramble and Horst Samulowitz and Dakuo Wang and Andrew R. Conn and Alexander Gray},
  title = {{A Formal Method for AutoML via ADMM}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5926},
}
Figure