Meta-learning for symbolic hyperparameter defaults

Pieter Gijsbers, Florian Pfisterer, Jan N. van Rijn, Bernd Bischl, Joaquin Vanschoren

TL;DR

Instead of using "one-size-fits-all" static default values for machine learning hyperparameters, this paper proposes a method to automatically "evolve" symbolic formulas that adapt to the characteristics of a dataset. By using genetic programming and meta-learning, the authors find data-dependent defaults (e.g., setting a parameter based on the number of features) that can outperform standard library defaults and even rival several iterations of random search—all at zero extra computational cost during deployment.

What Problem Does This Paper Address?

Most machine learning practitioners rely on "default" hyperparameters provided by libraries like scikit-learn or xgboost. While these defaults are convenient, they are often either static constants (like C=1.0 for an SVM) or simple hand-crafted heuristics (like sqrt(number_of_features) for Random Forests).

Optimizing hyperparameters (HPO) for every new dataset is the "gold standard" for performance, but it is computationally expensive and complex to implement in production pipelines. The authors address the gap between "dumb" static defaults and "expensive" full HPO. They ask: Can we automatically discover mathematical formulas that define the best defaults based on the properties of the data itself?

Methods

The authors treat the search for defaults as a Symbolic Regression problem, solved using Evolutionary Algorithms (specifically Genetic Programming).

  1. Meta-Feature Extraction: They identify simple characteristics of datasets (meta-features) such as the number of observations ($n$), number of features ($p$), and class balance.
  2. Surrogate Modeling: To avoid running thousands of ML experiments during the search, they build "surrogate models" (Random Forests) trained on a massive database of prior evaluations (OpenML). These surrogates predict how an algorithm would perform on a specific dataset with specific hyperparameters.
  3. Grammar-Based Evolution: They define a mathematical grammar (addition, log, powers, etc.) that the evolutionary algorithm uses to construct formulas. The algorithm (NSGA-II) evolves a population of formulas, selecting for those that minimize loss across many datasets while keeping the formulas relatively simple.
  4. Zero-Shot Deployment: Once a formula is found (e.g., $Hyperparameter = \log(n) / p$), it can be applied to any new dataset instantly without further optimization.

Datasets

The study utilizes the largest collection of metadata available at the time:

  • OpenML-CC18: A curated benchmark suite for machine learning.
  • AutoML Benchmark: A set of diverse datasets used to evaluate automated ML systems.
  • In total, they analyzed 106 to 119 datasets (depending on the algorithm) with sizes ranging from 100 to 130,000 observations and up to 10,000 features.

Key Insights and Ideas

  • The "Functional" Nature of Defaults: The core idea is that the "optimal" hyperparameter isn't a single point in space, but a function of the data's geometry. For example, the ideal kernel width for an SVM should naturally scale with the distance between data points.
  • Surrogates as a Sandbox: By using surrogate models, the authors can "simulate" the performance of millions of potential formulas in seconds, which would be impossible with real-world training.
  • Interpretability: Unlike "black-box" meta-learning (like using a neural network to predict HPs), symbolic defaults produce human-readable formulas. This allows researchers to inspect why a certain default works, potentially leading to new theoretical insights into algorithm behavior.
  • Complexity vs. Performance: The authors used multi-objective optimization to find a "Pareto front," balancing the accuracy of a default against the complexity of its formula.

Key Results

The authors tested their method across six major algorithms: SVM, Elastic Net, KNN, Decision Trees, Random Forests, and XGBoost.

  • Beating the Experts: The evolved symbolic defaults outperformed the hand-crafted defaults in popular libraries (like scikit-learn and mlr) in several scenarios.
  • Efficiency: Evolved defaults often performed as well as 8 to 16 iterations of Random Search. This is significant because the symbolic default is "free" (calculated instantly), whereas Random Search requires training the model 8-16 times.
  • Algorithm Specifics: For the SVM, the discovered symbolic formula for the $\gamma$ parameter was strikingly similar to the "scale" heuristic used in scikit-learn, but it was discovered entirely from data. For XGBoost, the method found that constant defaults were already quite strong, suggesting less dependency on dataset scale for its specific HPs.

Limitations and Future Work

  • Surrogate Dependency: The quality of the discovered formulas is capped by the accuracy of the surrogate models. If the surrogate doesn't capture a nuance of the ML algorithm, the evolved formula won't either.
  • Meta-Feature Selection: The current work uses a limited set of simple meta-features. Future work could include more complex "landmarking" features or information about the data distribution (e.g., skewness, kurtosis).
  • Categorical Parameters: The current grammar is best suited for continuous or integer-valued hyperparameters. Handling categorical choices (like choosing between different solvers) via symbolic formulas remains a challenge.
  • Sets of Defaults: The authors suggest moving toward "portfolios" of symbolic defaults—providing a small set of formulas to try rather than just one.

Summary updated

Abstract

Hyperparameter optimization in machine learning (ML) deals with the problem of empirically learning an optimal algorithm configuration from data, usually formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations that are expressed in terms of the properties of the dataset. This enables a much faster, but still data-dependent, configuration of the ML algorithm, compared to standard hyperparameter optimization approaches. In the past, symbolic and static default values have usually been obtained as hand-crafted heuristics. We propose an approach of learning such symbolic configurations as formulas of dataset properties from a large set of prior evaluations on multiple datasets by optimizing over a grammar of expressions using an evolutionary algorithm. We evaluate our method on surrogate empirical performance models as well as on real data across 6 ML algorithms on more than 100 datasets and demonstrate that our method indeed finds viable symbolic defaults.
BibTeX
@inproceedings{gijsbers2021meta,
  author = {Pieter Gijsbers and Florian Pfisterer and Jan N. van Rijn and Bernd Bischl and Joaquin Vanschoren},
  title = {{Meta-learning for symbolic hyperparameter defaults}},
  booktitle = {GECCO Companion},
  year = {2021},
  doi = {10.1145/3449726.3459532},
}

Loading related content...

Figure