TL;DR
Instead of using "one-size-fits-all" static default values for machine learning hyperparameters, this paper proposes a method to automatically "evolve" symbolic formulas that adapt to the characteristics of a dataset. By using genetic programming and meta-learning, the authors find data-dependent defaults (e.g., setting a parameter based on the number of features) that can outperform standard library defaults and even rival several iterations of random search—all at zero extra computational cost during deployment.
What Problem Does This Paper Address?
Most machine learning practitioners rely on "default" hyperparameters provided by libraries like scikit-learn or xgboost. While these defaults are convenient, they are often either static constants (like C=1.0 for an SVM) or simple hand-crafted heuristics (like sqrt(number_of_features) for Random Forests).
Optimizing hyperparameters (HPO) for every new dataset is the "gold standard" for performance, but it is computationally expensive and complex to implement in production pipelines. The authors address the gap between "dumb" static defaults and "expensive" full HPO. They ask: Can we automatically discover mathematical formulas that define the best defaults based on the properties of the data itself?
Methods
The authors treat the search for defaults as a Symbolic Regression problem, solved using Evolutionary Algorithms (specifically Genetic Programming).
- Meta-Feature Extraction: They identify simple characteristics of datasets (meta-features) such as the number of observations ($n$), number of features ($p$), and class balance.
- Surrogate Modeling: To avoid running thousands of ML experiments during the search, they build "surrogate models" (Random Forests) trained on a massive database of prior evaluations (OpenML). These surrogates predict how an algorithm would perform on a specific dataset with specific hyperparameters.
- Grammar-Based Evolution: They define a mathematical grammar (addition, log, powers, etc.) that the evolutionary algorithm uses to construct formulas. The algorithm (NSGA-II) evolves a population of formulas, selecting for those that minimize loss across many datasets while keeping the formulas relatively simple.
- Zero-Shot Deployment: Once a formula is found (e.g., $Hyperparameter = \log(n) / p$), it can be applied to any new dataset instantly without further optimization.
Datasets
The study utilizes the largest collection of metadata available at the time:
- OpenML-CC18: A curated benchmark suite for machine learning.
- AutoML Benchmark: A set of diverse datasets used to evaluate automated ML systems.
- In total, they analyzed 106 to 119 datasets (depending on the algorithm) with sizes ranging from 100 to 130,000 observations and up to 10,000 features.
Key Insights and Ideas
- The "Functional" Nature of Defaults: The core idea is that the "optimal" hyperparameter isn't a single point in space, but a function of the data's geometry. For example, the ideal kernel width for an SVM should naturally scale with the distance between data points.
- Surrogates as a Sandbox: By using surrogate models, the authors can "simulate" the performance of millions of potential formulas in seconds, which would be impossible with real-world training.
- Interpretability: Unlike "black-box" meta-learning (like using a neural network to predict HPs), symbolic defaults produce human-readable formulas. This allows researchers to inspect why a certain default works, potentially leading to new theoretical insights into algorithm behavior.
- Complexity vs. Performance: The authors used multi-objective optimization to find a "Pareto front," balancing the accuracy of a default against the complexity of its formula.
Key Results
The authors tested their method across six major algorithms: SVM, Elastic Net, KNN, Decision Trees, Random Forests, and XGBoost.
- Beating the Experts: The evolved symbolic defaults outperformed the hand-crafted defaults in popular libraries (like
scikit-learnandmlr) in several scenarios. - Efficiency: Evolved defaults often performed as well as 8 to 16 iterations of Random Search. This is significant because the symbolic default is "free" (calculated instantly), whereas Random Search requires training the model 8-16 times.
- Algorithm Specifics: For the SVM, the discovered symbolic formula for the $\gamma$ parameter was strikingly similar to the "scale" heuristic used in
scikit-learn, but it was discovered entirely from data. For XGBoost, the method found that constant defaults were already quite strong, suggesting less dependency on dataset scale for its specific HPs.
Limitations and Future Work
- Surrogate Dependency: The quality of the discovered formulas is capped by the accuracy of the surrogate models. If the surrogate doesn't capture a nuance of the ML algorithm, the evolved formula won't either.
- Meta-Feature Selection: The current work uses a limited set of simple meta-features. Future work could include more complex "landmarking" features or information about the data distribution (e.g., skewness, kurtosis).
- Categorical Parameters: The current grammar is best suited for continuous or integer-valued hyperparameters. Handling categorical choices (like choosing between different solvers) via symbolic formulas remains a challenge.
- Sets of Defaults: The authors suggest moving toward "portfolios" of symbolic defaults—providing a small set of formulas to try rather than just one.
Summary updated