TL;DR
Think you’re giving your Bayesian Optimization (BO) a "head start" by initializing it near library default hyperparameters? Think again. This study demonstrates that centering BO initialization around expert-chosen defaults (like those in scikit-learn) provides no statistically significant advantage in convergence speed or final model quality over simple uniform random sampling.
What Problem Does This Paper Address?
Bayesian Optimization is the gold standard for tuning hyperparameters because it is "sample efficient"—it finds good settings with very few expensive model evaluations. However, every BO pipeline needs a starting point.
The prevailing wisdom is to start with a few points chosen uniformly at random. But many researchers and engineers have long suspected that we are leaving "expert knowledge" on the table. Popular libraries like scikit-learn, XGBoost, and LightGBM ship with default hyperparameters carefully selected by developers to perform reasonably well across many tasks. The authors of this paper ask a simple but critical question: Does starting our search near these "safe" defaults actually help the optimizer find the global optimum faster?
Methods
The researchers formalized "default-informed initialization" by drawing initial points from a truncated Gaussian distribution centered at the library defaults. They compared this against a standard uniform random baseline.
To ensure the results weren't just a fluke of one specific tool or model, they built a massive experimental grid:
- BO Back-ends: BoTorch (GP-based), Scikit-Optimize (GP-based), and Optuna (Tree-structured Parzen Estimator).
- Model Families: Random Forests, Support Vector Machines (SVM), and Multilayer Perceptrons (MLP).
- Initialization Strategies:
- Default: Starting with the exact deterministic default point.
- Sample: Drawing 3–5 points from the truncated Gaussian.
- Concentration Parameter ($\lambda$): They varied how "tightly" the initial points clustered around the default to see if a very specific or a very broad "prior" made a difference.
Datasets
The study used five benchmark datasets covering both classification and regression:
- AmazonAcc: Binary access prediction.
- Letter: Multiclass character recognition.
- Higgs: Particle physics (large and noisy).
- BikeSharing: Regression on rental demand.
- Airlines10M: Airline delay prediction.
To keep the thousands of required runs computationally feasible, all datasets were subsampled to 1,000 instances.
Key Insights and Ideas
- "Safe" is not "Informative": The authors argue that while defaults are engineered to be "safe" (they won't fail catastrophically), they don't actually encode the shape of the loss landscape. They are a shortcut for deployment, not a map for optimization.
- The Risk of "Conformist" Behavior: Clumping initial observations near the default causes the Gaussian Process surrogate to become overconfident in that small neighborhood. This can actually suppress exploration of distant regions where the true global optimum might reside.
- Transient vs. Asymptotic Gains: Any benefit from starting near a default is "transient." While the very first evaluation might look better than a random point, the optimizer quickly closes that gap. By the time you reach the end of your tuning budget, the starting point is irrelevant.
Key Results
The results were, in the authors' words, "unequivocal." * No Statistical Edge: Using one-sided binomial tests, the researchers found no significant advantage for default-informed initialization. * High P-Values: The p-values for their hypotheses ranged from 0.141 to 0.908, far above the standard 0.05 threshold for significance. * Sensitivity Analysis: Even when they made the initialization extremely tight around the defaults ($\lambda = 0.05$), the final performance remained indistinguishable from random sampling. * The Verdict: Random initialization provides broader coverage of the search space, giving the surrogate model a more informative "global" foundation from the very first few steps.
Limitations and Future Work
The authors acknowledge a few caveats:
- Subsampling: Reducing datasets to 1,000 rows might change the relationship between hyperparameters and performance compared to "big data" scenarios.
- Model Scope: The study focused on classical ML models. Future work is needed to see if these findings hold for deep learning, where search spaces are much larger and evaluation costs are higher.
- Acquisition Functions: The study used Expected Improvement (EI) exclusively. Other functions like Thompson Sampling or Knowledge Gradient might interact differently with the initialization strategy.
The takeaway for practitioners? Treat hyperparameter tuning as a purely data-driven process. Don't rely on heuristics or "expert" defaults to save you time; stick to principled, exploratory search strategies from the start.
Summary updated