Abstract
In machine learning, hyperparameter optimization is a challenging task that is usually approached by experienced practitioners or in a computationally expensive brute-force manner such as grid-search. Therefore, recent research proposes to use observed hyperparameter performance on already solved problems (i.e. data sets) in order to speed up the search for promising hyperparameter configurations in the sequential model based optimization framework.
Introduction
Finding the best hyperparameters for a machine learning model—such as the learning rate of a neural network or the $C$ value of a Support Vector Machine (SVM)—is notoriously difficult. Traditionally, practitioners rely on "brute-force" methods like grid search or random search, which are computationally expensive and treat every new dataset as a blank slate. This paper introduces Factorized Multilayer Perceptrons (FMLP) as a powerful surrogate model for hyperparameter optimization (HPO) that leverages meta-learning to transfer knowledge from past experiments.
Sequential Model-Based Optimization (SMBO)
The authors focus on the SMBO framework, where a "surrogate model" is trained to predict a model's performance based on its hyperparameters. Standard surrogates, such as Gaussian Processes or Random Forests, often struggle to generalize across different datasets. FMLP aims to overcome this by effectively performing meta-learning: using performance data from previous datasets to predict optimal configurations for new ones.
Methods
The paper proposes three main advancements for surrogate modeling:
- Multilayer Perceptrons (MLP) as Surrogates: Unlike standard linear models, MLPs can capture the highly non-linear "response surfaces" of hyperparameter performance.
- Factorized MLPs (FMLP): This core innovation explicitly models interactions. While a standard MLP might eventually learn how hyperparameters interact with dataset features, the FMLP forces this learning to happen via a Factorization Machine layer at the input. This layer models second-order interactions between hyperparameters and dataset characteristics.
- Uncertainty Estimation: SMBO requires balancing exploration and exploitation. The authors compared a complex analytical approach (using the Inverse Hessian/Laplace approximation) against a simpler Ensemble approach (training 100 MLPs with different initializations).
The training of these models is handled via a specialized backpropagation algorithm that accounts for the factorized components, as detailed in the algorithm below.
Datasets and Evaluation
The authors created two new meta-datasets based on 25 classification tasks from the UCI repository:
- AdaBoost Meta-dataset: Contains 108 hyperparameter configurations per dataset, varying iterations ($I$) and product terms ($M$).
- SVM Meta-dataset: A more complex "cross-model" dataset with 288 configurations, including kernel choice (Linear, Polynomial, RBF) and associated parameters ($C$, degree $d$, and width $\gamma$).
Key Insights
- Explicit vs. Implicit Modeling: Explicitly modeling the interaction between "Dataset ID" and "Hyperparameter Value" via factorization (FMLP) led to significantly better performance than a standard MLP.
- The "Hyperbolic Paraboloid" Problem: Plain Factorization Machines (without MLP layers) are too simple for HPO; they create "saddle" shapes that cannot represent the complex peaks and valleys of real hyperparameter response surfaces.
- Ensembles for Uncertainty: Surprisingly, the simple ensemble method (MLPE) outperformed the mathematically rigorous Hessian-based approach (MLPH). The Hessian approximation proved too noisy in the early stages of HPO when data is sparse.
Results and Conclusion
In leave-one-out cross-validation, the FMLP achieved the lowest Root Mean Squared Error (RMSE) for both SVM (0.0550) and AdaBoost (0.0377), outperforming Random Forests and Support Vector Regression. In full SMBO settings, FMLP reached optimal configurations faster than established baselines like SMAC++ and SCoT.
While the ensemble approach is effective, its scalability remains a concern for larger meta-datasets. Future work aims to expand this into a "universal" HPO surrogate capable of handling a massive library of different base models simultaneously by learning a shared latent feature space.
Summary updated