Transfer Learning for Bayesian HPO with End-to-End Meta-Features

Hadi Samer Jomaa, Sebastian Pineda Arango, Lars Schmidt-Thieme, Josif GrabockaICLR 2022 · OpenReview

Abstract

Hyperparameter optimization (HPO) is a crucial component of deploying machine learning models, however, it remains an open problem due to the resource-constrained number of possible hyperparameter evaluations. As a result, prior work focus on exploring the direction of transfer learning for tackling the sample inefficiency of HPO. In contrast to existing approaches, we propose a novel Deep Kernel Gaussian Process surrogate with Landmark Meta-features (DKLM) that can be jointly meta-trained on a set of source tasks and then transferred efficiently on a new (unseen) target task. We design DKLM to capture the similarity between hyperparameter configurations with an end-to-end meta-feature network that embeds the set of evaluated configurations and their respective performance. As a result, our novel DKLM can learn contextualized dataset-specific similarity representations for hyperparameter configurations. We experimentally validate the performance of DKLM in a wide range of HPO meta-datasets from OpenML and demonstrate the empirical superiority of our method against a series of state-of-the-art baselines.
BibTeX
@inproceedings{jomaa2022transfer,
  author = {Hadi Samer Jomaa and Sebastian Pineda Arango and Lars Schmidt-Thieme and Josif Grabocka},
  title = {{Transfer Learning for Bayesian HPO with End-to-End Meta-Features}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=wronZ3Mx_d},
}

Loading related content...

Figure