Bayesian Optimization Meets Self-Distillation

HyunJae Lee, Heon Song, Hyeonsoo Lee, Gihyeon Lee, Suyeong Park, Donggeun YooICCV 2023 · DOI · arXiv

Abstract

Hyperparameter Optimization (HPO) is a critical yet computationally expensive component of deep learning. Conventional Bayesian Optimization (BO) methods iteratively search the hyperparameter space but discard the trained weights of each trial, retaining only the final performance score. This "knowledge waste" represents a significant inefficiency. We introduce BOSS (Bayesian Optimization meets Self-diStillation), a novel framework that treats high-performing models from previous trials as "teachers" to initialize and guide the training of new models. By integrating BO with self-distillation and employing an asymmetric initialization strategy, BOSS achieves superior generalization and performance across diverse benchmarks, including image classification, noisy-label learning, and medical imaging.

Introduction

In the world of deep learning, finding the perfect set of hyperparameters—learning rates, batch sizes, weight decay—is often a dark art. Researchers typically turn to Bayesian Optimization (BO) to automate this search. While BO is effective at suggesting which "knobs" to turn next, it has a glaring inefficiency: it throws away the actual trained weights of every model it tests, keeping only the final performance score.

BOSS (Bayesian Optimization meets Self-diStillation) is a framework designed to stop this knowledge waste. By treating previous training trials not just as data points, but as potential teachers, BOSS pushes the boundaries of what a model can learn during the optimization process.

The BOSS Framework

The BOSS framework operates in two distinct phases to ensure knowledge persistence across the optimization cycle:

  1. Phase 1: Warm-up: This phase utilizes standard Bayesian Optimization. The system explores the hyperparameter space using a surrogate model, such as the Tree-structured Parzen Estimator (TPE). It completes a set number of trials ($W$) to build an initial library of trained models and performance observations.
  2. Phase 2: BOSS Training: In this phase, the system leverages the accumulated knowledge. For every new trial, BOSS identifies the "Top-K" best-performing models from all previous trials. It then randomly selects two models from this elite group: one to initialize the weights of the new "student" model (a warm-start) and another to act as the "teacher."
The BOSS framework iteratively combines Bayesian Optimization with Self-Distillation across trials.

The Power of Asymmetric Initialization

A key finding of the BOSS framework is the importance of asymmetric initialization. The authors discovered that using different previous models for the student and the teacher is vital. This provides a diversity of "knowledge" that helps the student model aggregate diverse features from multiple high-performing ancestors, ultimately helping it escape local optima and find flatter minima.

Experimental Results

The researchers evaluated BOSS across a rigorous gauntlet of tests, including general computer vision (CIFAR-10/100, Tiny ImageNet), robustness to noisy labels, semi-supervised learning, and medical image analysis (breast cancer classification and nuclei segmentation).

BOSS consistently outperformed standalone BO and Self-Distillation (SD) across every benchmark. For instance, on CIFAR-100, BOSS achieved 77.69% accuracy, compared to 76.48% for standard BO and 74.43% for the baseline. In medical imaging, BOSS boosted the Area Under the Curve (AUC) on the CMMD dataset to 85.81, a significant jump from the 79.22 baseline.

Top-1 accuracy comparison across trials for BOSS and various baseline methods.

Key Insights and Efficiency

  • Knowledge Persistence: BOSS transforms BO from a memoryless search for scalars into a persistent accumulation of model weights. Each trial "stands on the shoulders" of the best previous versions.
  • Efficiency over Size: In experiments, a smaller ResNet-20 trained with BOSS outperformed a much larger ResNet-56 trained with standard methods. This suggests that smarter optimization can be more effective than simply increasing parameter counts.
  • Versatility: BOSS is not tied to one specific algorithm; it works effectively with different BO surrogate models (like Gaussian Processes) and various distillation techniques.

Limitations and Future Work

While BOSS is powerful, it introduces a storage overhead by requiring a library of "Top-K" model weights, which could be challenging for massive models like LLMs. Additionally, it introduces new hyperparameters, such as the number of warm-up trials ($W$) and the candidate pool size ($K$). Future work aims to extend BOSS to other tasks like object detection and generative modeling.

Summary updated

BibTeX
@inproceedings{lee2023bayesian,
  author = {HyunJae Lee and Heon Song and Hyeonsoo Lee and Gihyeon Lee and Suyeong Park and Donggeun Yoo},
  title = {{Bayesian Optimization Meets Self-Distillation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00163},
}
Figure