Abstract
Hyperparameter optimization (HPO) in machine learning (ML) typically involves empirically learning an optimal algorithm configuration from data, often formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations expressed in terms of dataset properties. This approach enables faster, data-dependent configuration compared to standard HPO. By optimizing over a grammar of expressions using an evolutionary algorithm and evaluating on surrogate empirical performance models, we demonstrate that our method finds viable symbolic defaults across 6 ML algorithms on more than 100 datasets.
Introduction: Beyond Static Defaults
Most machine learning practitioners rely on library defaults (e.g., from scikit-learn or xgboost) which are typically static constants or simple heuristics. While full hyperparameter optimization (HPO) yields better performance, it is computationally expensive. This research explores a middle ground: automatically discovering mathematical formulas that define optimal defaults based on the specific properties of a dataset.
Methodology: Evolving Symbolic Formulas
The authors frame the search for defaults as a Symbolic Regression problem. Instead of predicting a single value, the goal is to find a function $f(\text{meta-features}) = \text{Hyperparameter}$.
1. Meta-Feature Extraction and Surrogate Modeling
To facilitate the search without the prohibitive cost of running millions of ML experiments, the authors utilize surrogate models. These models are trained on the OpenML-CC18 benchmark and the AutoML Benchmark, covering over 100 datasets. The surrogates predict the performance of an algorithm given a dataset's meta-features (like number of observations $n$ and features $p$) and a specific hyperparameter configuration.
As shown in the validation plots, the surrogate models achieve high rank correlation (Spearman's $\rho$ and Kendall's $\tau$), ensuring that the evolutionary search is guided by accurate performance estimates.
2. Evolutionary Search
Using the NSGA-II evolutionary algorithm, the researchers evolve a population of formulas defined by a mathematical grammar. The optimization process balances two objectives:
- Performance: Minimizing the predicted loss across the meta-dataset.
- Simplicity: Keeping formulas interpretable and avoiding overfitting.
Key Results and Benchmarking
The evolved symbolic defaults were tested against standard library defaults (scikit-learn, mlr) and random search baselines.
The Critical Difference (CD) diagram illustrates the relative performance of different strategies. Key findings include:
- Superiority over Library Defaults: Symbolic defaults consistently outperformed the static defaults provided by popular ML libraries.
- Efficiency Gains: In many cases, the "zero-shot" symbolic default performed as well as 8 to 16 iterations of random search. This represents a significant efficiency gain, as the symbolic formula is computed instantly, whereas random search requires multiple training cycles.
- Interpretability: The discovered formulas often mirrored known heuristics. For example, the evolved formula for SVM's $\gamma$ parameter closely resembled the "scale" heuristic, validating the method's ability to recover meaningful patterns from data.
Conclusion
This work demonstrates that symbolic defaults are a powerful tool for automated machine learning. By moving from static constants to data-dependent formulas, practitioners can achieve near-HPO performance at zero additional computational cost during deployment. Future work will explore expanding the meta-feature set and developing "portfolios" of symbolic defaults to further improve robustness across diverse data distributions.
Summary updated