Abstract
We study the AutoML problem of automatically configuring machine learning pipelines by jointly selecting algorithms and their appropriate hyper-parameters for all steps in supervised learning pipelines. This black-box (gradient-free) optimization with mixed integer & continuous variables is a challenging problem. We propose a novel AutoML scheme by leveraging the alternating direction method of multipliers (ADMM). The proposed framework is able to (i) decompose the optimization problem into easier sub-problems that have a reduced number of variables and circumvent the challenge of mixed variable categories, and (ii) incorporate black-box constraints along-side the black-box optimization objective. We empirically evaluate the flexibility, effectiveness, and unique capability of our proposed scheme on a collection of binary classification data sets from UCI ML & OpenML repositories. We observe that on an average our framework provides significant gains in comparison to other AutoML frameworks (Auto-sklearn & TPOT), highlighting the practical advantages of this framework.
Introduction
Building an effective machine learning pipeline involves solving the Combined Algorithm Selection and Hyper-parameter optimization (CASH) problem. This task is inherently difficult due to the presence of mixed variables (continuous and discrete), the black-box nature of the objective function (requiring slow training/testing cycles), and the need to satisfy real-world constraints such as latency and fairness. Traditional AutoML tools often focus solely on accuracy, neglecting these critical operational constraints.
Methodology: Decomposing the CASH Problem
The proposed framework adapts the Alternating Direction Method of Multipliers (ADMM) to the non-convex, black-box domain of AutoML. By employing a "divide and conquer" strategy, the complex pipeline configuration task is decomposed into three iterative sub-problems:
- Continuous Hyper-parameter Optimization ($\theta$-min): Optimizes continuous parameters for selected algorithms using Bayesian Optimization (BO).
- Integer Projection ($\delta$-min): Handles integer-based hyper-parameters via Euclidean projection (rounding to the nearest valid integer).
- Algorithm Selection ($z$-min): Selects the optimal combination of algorithms, modeled as a Combinatorial Multi-Armed Bandit (CMAB) problem solved with Thompson Sampling.
To enhance efficiency, the authors introduce Adaptive ADMM, which uses lower-precision internal solvers in early iterations to accelerate the search process, gradually increasing precision as the search converges. This approach can lead to temporal speedups of up to 150x in certain benchmarks.
Constraint-Aware AutoML
A distinguishing feature of this ADMM-based approach is its ability to handle "black-box" constraints. By integrating auxiliary variables, constraints like inference latency or fairness metrics can be incorporated directly into the optimization loop. This allows the framework to find high-performing models that are also deployable in regulated or resource-constrained environments.
Experimental Evaluation
The framework was tested on 30 binary classification datasets from UCI, OpenML, and Kaggle, as well as the Home Credit Default Risk (HCDR) dataset for constraint testing.
Key Results
The ADMM framework demonstrated superior performance and efficiency across the benchmark suite:
- Performance: ADMM achieved the best results on 50% of the datasets, outperforming Auto-sklearn (27%) and TPOT (20%).
- Efficiency: The framework achieved over a 10x speedup and 10% improvement in objective values in many cases compared to joint optimization baselines.
- Constraint Satisfaction: On the HCDR dataset, ADMM successfully identified pipelines that met strict fairness and latency requirements, whereas unconstrained methods typically failed to find valid solutions.
Conclusion and Future Work
While the framework lacks the theoretical convergence guarantees of convex ADMM, its empirical success is notable. The decomposition approach significantly reduces the search space complexity, leading to massive temporal speedups. Future work includes extending the framework to handle flexible pipeline topologies and expanding the library of supported algorithms. The ability to treat operational constraints as first-class citizens marks a significant step toward making AutoML tools more practical for real-world applications.
Summary updated