Meta-Learning for Symbolic Hyperparameter Defaults

Pieter Gijsbers, Florian Pfisterer, Jan N. van Rijn, Bernd Bischl, Joaquin Vanschoren

Abstract

Hyperparameter optimization (HPO) in machine learning (ML) typically involves empirically learning an optimal algorithm configuration from data, often formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations expressed in terms of dataset properties. This approach enables faster, data-dependent configuration compared to standard HPO. By optimizing over a grammar of expressions using an evolutionary algorithm and evaluating on surrogate empirical performance models, we demonstrate that our method finds viable symbolic defaults across 6 ML algorithms on more than 100 datasets.

Introduction: Beyond Static Defaults

Most machine learning practitioners rely on library defaults (e.g., from scikit-learn or xgboost) which are typically static constants or simple heuristics. While full hyperparameter optimization (HPO) yields better performance, it is computationally expensive. This research explores a middle ground: automatically discovering mathematical formulas that define optimal defaults based on the specific properties of a dataset.

Methodology: Evolving Symbolic Formulas

The authors frame the search for defaults as a Symbolic Regression problem. Instead of predicting a single value, the goal is to find a function $f(\text{meta-features}) = \text{Hyperparameter}$.

1. Meta-Feature Extraction and Surrogate Modeling

To facilitate the search without the prohibitive cost of running millions of ML experiments, the authors utilize surrogate models. These models are trained on the OpenML-CC18 benchmark and the AutoML Benchmark, covering over 100 datasets. The surrogates predict the performance of an algorithm given a dataset's meta-features (like number of observations $n$ and features $p$) and a specific hyperparameter configuration.

Surrogate model validation showing high correlation between predicted and actual performance.

As shown in the validation plots, the surrogate models achieve high rank correlation (Spearman's $\rho$ and Kendall's $\tau$), ensuring that the evolutionary search is guided by accurate performance estimates.

Using the NSGA-II evolutionary algorithm, the researchers evolve a population of formulas defined by a mathematical grammar. The optimization process balances two objectives:

  • Performance: Minimizing the predicted loss across the meta-dataset.
  • Simplicity: Keeping formulas interpretable and avoiding overfitting.

Key Results and Benchmarking

The evolved symbolic defaults were tested against standard library defaults (scikit-learn, mlr) and random search baselines.

Critical Difference diagram comparing symbolic defaults to library defaults and random search iterations.

The Critical Difference (CD) diagram illustrates the relative performance of different strategies. Key findings include:

  • Superiority over Library Defaults: Symbolic defaults consistently outperformed the static defaults provided by popular ML libraries.
  • Efficiency Gains: In many cases, the "zero-shot" symbolic default performed as well as 8 to 16 iterations of random search. This represents a significant efficiency gain, as the symbolic formula is computed instantly, whereas random search requires multiple training cycles.
  • Interpretability: The discovered formulas often mirrored known heuristics. For example, the evolved formula for SVM's $\gamma$ parameter closely resembled the "scale" heuristic, validating the method's ability to recover meaningful patterns from data.

Conclusion

This work demonstrates that symbolic defaults are a powerful tool for automated machine learning. By moving from static constants to data-dependent formulas, practitioners can achieve near-HPO performance at zero additional computational cost during deployment. Future work will explore expanding the meta-feature set and developing "portfolios" of symbolic defaults to further improve robustness across diverse data distributions.

Summary updated

Abstract

Hyperparameter optimization in machine learning (ML) deals with the problem of empirically learning an optimal algorithm configuration from data, usually formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations that are expressed in terms of the properties of the dataset. This enables a much faster, but still data-dependent, configuration of the ML algorithm, compared to standard hyperparameter optimization approaches. In the past, symbolic and static default values have usually been obtained as hand-crafted heuristics. We propose an approach of learning such symbolic configurations as formulas of dataset properties from a large set of prior evaluations on multiple datasets by optimizing over a grammar of expressions using an evolutionary algorithm. We evaluate our method on surrogate empirical performance models as well as on real data across 6 ML algorithms on more than 100 datasets and demonstrate that our method indeed finds viable symbolic defaults.
BibTeX
@inproceedings{gijsbers2021meta,
  author = {Pieter Gijsbers and Florian Pfisterer and Jan N. van Rijn and Bernd Bischl and Joaquin Vanschoren},
  title = {{Meta-Learning for Symbolic Hyperparameter Defaults}},
  booktitle = {GECCO Companion},
  year = {2021},
  doi = {10.1145/3449726.3459532},
}
Figure