Introduction
Bayesian Optimization (BayesOpt) is widely considered the gold standard for tuning machine learning hyperparameters. However, standard BayesOpt is surprisingly "wasteful"—it spends hours or days testing different configurations only to throw away every result except the single best one. This paper introduces EB-Hyp, a hierarchical empirical Bayes approach that averages over multiple hyperparameter settings instead of picking just one. This method leads to better generalization and can even eliminate the need for a separate validation dataset.
The Problem with Standard BayesOpt
When tuning a machine learning model, practitioners typically use BayesOpt to find the optimal hyperparameters (e.g., learning rate, number of layers). This process is computationally expensive and follows a strict "winner-takes-all" philosophy: find the configuration that performs best on a validation set and discard the rest.
The authors identify three major flaws in this standard pipeline:
- Wastefulness: Every "sub-optimal" hyperparameter setting tested produces a trained model that contains useful information about the data. Throwing them away wastes expensive computation.
- Overfitting: If the hyperparameter space is large or the validation set is small, BayesOpt can easily overfit the validation data. The "best" setting on validation might actually perform poorly on truly unseen test data.
- Complexity: Maintaining a three-way split (train/validation/test) adds complexity and reduces the amount of data available for actual training.
The EB-Hyp Approach
To address these issues, the authors propose Empirical Bayes for hyperparameter averaging (EB-Hyp). Instead of treating hyperparameter tuning as an optimization problem ($\text{argmax}$), they treat it as an integration problem (marginalization).
Key technical components include:
- Hierarchical Empirical Bayes: The authors introduce "hyper-hyperparameters" ($\lambda$) to define a prior over the hyperparameters. They then marginalize over the hyperparameters to create a predictive distribution that acts as a weighted average of many models.
- Performance Model Sampling: To decide which hyperparameters to test next, they use a generalization of Thompson Sampling. They maintain a Gaussian Process (GP) to model the performance of the hyperparameter space and draw samples from the GP's posterior to guide exploration.
- The Train-Marginalize-Test Pipeline: Unlike the standard "Train-Validate-Test" workflow, EB-Hyp allows practitioners to train models on various hyperparameters and then marginalize them out. This effectively uses the uncertainty about which hyperparameter is best to create a more robust, ensemble-like prediction.
Key Insights: Overfitting and Ensembling
A major "aha" moment in the paper is the demonstration of validation overfitting. The hyperparameter setting that looks best on validation data often isn't the best on test data.
By averaging over the top-performing models rather than picking the single best one, EB-Hyp acts as a form of regularization. It essentially creates an ensemble of models "for free." Unlike traditional ensembling techniques like Bagging or Boosting, these models are already being trained as part of the standard search process; EB-Hyp simply changes how we combine them at the end. Consequently, because EB-Hyp marginalizes over hyperparameters, it removes the strict requirement for a separate validation set, freeing up more data for training.
Experimental Results
The authors evaluated EB-Hyp on two distinct probabilistic models: Latent Dirichlet Allocation (LDA) on the 20 Newsgroups dataset, and a Deep Latent Gaussian Model (DLGM) on the Labeled Faces in the Wild (LFW) dataset.
The experiments demonstrated that EB-Hyp consistently outperformed standard BayesOpt and random search:
- LDA (20 Newsgroups): EB-Hyp without a validation set achieved a +1.60% improvement in predictive log-likelihood over BayesOpt with a validation set.
- DLGM (LFW): EB-Hyp without a validation set showed a significant +7.02% improvement over the BayesOpt baseline.
- Robustness: In both cases, the "without validation" version of EB-Hyp performed better than the version "with validation," proving that the marginalization approach effectively utilizes all available data without succumbing to overfitting.
Limitations and Future Work
While powerful, EB-Hyp has a few limitations. The method relies on the assumption that the marginal likelihood is relatively "smooth" across the hyperparameter space; sudden, catastrophic jumps in model behavior could challenge the Gaussian Process performance model. Additionally, while EB-Hyp is more efficient in its use of results, it doesn't inherently speed up the training of individual models—the "inner loop" of training remains the primary computational bottleneck. Finally, the authors used a nonparametric hyperprior (averaging the top 10% of samples) as a weighting heuristic, noting that more sophisticated ways to optimize the hyper-hyperparameters could be explored in future research.
Summary updated