Default Machine Learning Hyperparameters Do Not Provide Informative Initialization for Bayesian Optimization

Nicolás Villagrán Prieto, Eduardo C. Garrido-MerchánarXiv 2026 · DOI · arXiv

TL;DR

Think you’re giving your Bayesian Optimization (BO) a "head start" by initializing it near library default hyperparameters? Think again. This study demonstrates that centering BO initialization around expert-chosen defaults (like those in scikit-learn) provides no statistically significant advantage in convergence speed or final model quality over simple uniform random sampling.

What Problem Does This Paper Address?

Bayesian Optimization is the gold standard for tuning hyperparameters because it is "sample efficient"—it finds good settings with very few expensive model evaluations. However, every BO pipeline needs a starting point.

The prevailing wisdom is to start with a few points chosen uniformly at random. But many researchers and engineers have long suspected that we are leaving "expert knowledge" on the table. Popular libraries like scikit-learn, XGBoost, and LightGBM ship with default hyperparameters carefully selected by developers to perform reasonably well across many tasks. The authors of this paper ask a simple but critical question: Does starting our search near these "safe" defaults actually help the optimizer find the global optimum faster?

Methods

The researchers formalized "default-informed initialization" by drawing initial points from a truncated Gaussian distribution centered at the library defaults. They compared this against a standard uniform random baseline.

To ensure the results weren't just a fluke of one specific tool or model, they built a massive experimental grid:

  • BO Back-ends: BoTorch (GP-based), Scikit-Optimize (GP-based), and Optuna (Tree-structured Parzen Estimator).
  • Model Families: Random Forests, Support Vector Machines (SVM), and Multilayer Perceptrons (MLP).
  • Initialization Strategies:
    • Default: Starting with the exact deterministic default point.
  • Sample: Drawing 3–5 points from the truncated Gaussian.
  • Concentration Parameter ($\lambda$): They varied how "tightly" the initial points clustered around the default to see if a very specific or a very broad "prior" made a difference.

Datasets

The study used five benchmark datasets covering both classification and regression:

  1. AmazonAcc: Binary access prediction.
  2. Letter: Multiclass character recognition.
  3. Higgs: Particle physics (large and noisy).
  4. BikeSharing: Regression on rental demand.
  5. Airlines10M: Airline delay prediction.

To keep the thousands of required runs computationally feasible, all datasets were subsampled to 1,000 instances.

Key Insights and Ideas

  • "Safe" is not "Informative": The authors argue that while defaults are engineered to be "safe" (they won't fail catastrophically), they don't actually encode the shape of the loss landscape. They are a shortcut for deployment, not a map for optimization.
  • The Risk of "Conformist" Behavior: Clumping initial observations near the default causes the Gaussian Process surrogate to become overconfident in that small neighborhood. This can actually suppress exploration of distant regions where the true global optimum might reside.
  • Transient vs. Asymptotic Gains: Any benefit from starting near a default is "transient." While the very first evaluation might look better than a random point, the optimizer quickly closes that gap. By the time you reach the end of your tuning budget, the starting point is irrelevant.

Key Results

The results were, in the authors' words, "unequivocal." * No Statistical Edge: Using one-sided binomial tests, the researchers found no significant advantage for default-informed initialization. * High P-Values: The p-values for their hypotheses ranged from 0.141 to 0.908, far above the standard 0.05 threshold for significance. * Sensitivity Analysis: Even when they made the initialization extremely tight around the defaults ($\lambda = 0.05$), the final performance remained indistinguishable from random sampling. * The Verdict: Random initialization provides broader coverage of the search space, giving the surrogate model a more informative "global" foundation from the very first few steps.

Limitations and Future Work

The authors acknowledge a few caveats:

  • Subsampling: Reducing datasets to 1,000 rows might change the relationship between hyperparameters and performance compared to "big data" scenarios.
  • Model Scope: The study focused on classical ML models. Future work is needed to see if these findings hold for deep learning, where search spaces are much larger and evaluation costs are higher.
  • Acquisition Functions: The study used Expected Improvement (EI) exclusively. Other functions like Thompson Sampling or Knowledge Gradient might interact differently with the initialization strategy.

The takeaway for practitioners? Treat hyperparameter tuning as a purely data-driven process. Don't rely on heuristics or "expert" defaults to save you time; stick to principled, exploratory search strategies from the start.

Summary updated

Abstract

Bayesian Optimization (BO) is a standard tool for hyperparameter tuning thanks to its sample efficiency on expensive black-box functions. While most BO pipelines begin with uniform random initialization, default hyperparameter values shipped with popular ML libraries such as scikit-learn encode implicit expert knowledge and could serve as informative starting points that accelerate convergence. This hypothesis, despite its intuitive appeal, has remained largely unexamined. We formalize the idea by initializing BO with points drawn from truncated Gaussian distributions centered at library defaults and compare the resulting trajectories against a uniform-random baseline. We conduct an extensive empirical evaluation spanning three BO back-ends (BoTorch, Optuna, Scikit-Optimize), three model families (Random Forests, Support Vector Machines, Multilayer Perceptrons), and five benchmark datasets covering classification and regression tasks. Performance is assessed through convergence speed and final predictive quality, and statistical significance is determined via one-sided binomial tests. Across all conditions, default-informed initialization yields no statistically significant advantage over purely random sampling, with p-values ranging from 0.141 to 0.908. A sensitivity analysis on the prior variance confirms that, while tighter concentration around the defaults improves early evaluations, this transient benefit vanishes as optimization progresses, leaving final performance unchanged. Our results provide no evidence that default hyperparameters encode useful directional information for optimization. We therefore recommend that practitioners treat hyperparameter tuning as an integral part of model development and favor principled, data-driven search strategies over heuristic reliance on library defaults.
BibTeX
@misc{prieto2026default,
  author = {Nicolás Villagrán Prieto and Eduardo C. Garrido-Merchán},
  title = {{Default Machine Learning Hyperparameters Do Not Provide Informative Initialization for Bayesian Optimization}},
  year = {2026},
  eprint = {2602.08774},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.08774},
}

Loading related content...

Figure