Efficient Transfer Learning Method for Automatic Hyperparameter Tuning

Dani Yogatama, Gideon MannAISTATS 2014 · OpenReview

Abstract

We propose a fast and effective algorithm for automatic hyperparameter tuning that can generalize across datasets. Our method is an instance of sequential model-based optimization (SMBO) that transforms performance metrics into a standardized space, creating a common response surface. This allows for efficient transfer learning with linear computational complexity.

Introduction

Hyperparameter tuning is often described as a "black art," requiring extensive trial-and-error to find optimal settings for learning rates or regularization. Sequential Model-Based Optimization (SMBO) has emerged as a powerful tool for this task by building surrogate models of the search space. However, standard SMBO typically restarts for every new dataset, ignoring valuable information from previous experiments.

Transfer learning aims to bridge this gap, but it faces two primary hurdles: the lack of comparable metrics across different datasets and the high computational cost of existing ranking-based transfer methods. This paper introduces a streamlined approach that addresses both issues through metric normalization and a scalable Gaussian Process (GP) framework.

Methodology: The Common Response Surface

The authors' primary innovation is the Standardized Response Surface. Instead of predicting raw accuracy, which varies wildly between "easy" and "hard" datasets, the model predicts the deviation from the per-dataset mean: $y = (accuracy - \mu) / \sigma$. This normalization maps all datasets onto a comparable scale, revealing a shared geography of hyperparameter performance.

Mapping individual dataset accuracy surfaces to a common response surface.

To effectively utilize this surface, the method employs Multiple Kernel Learning (MKL). The surrogate model uses a combination of two kernels:

  1. Squared Exponential Kernel: Captures the specific nuances of the current dataset.
  2. Nearest Neighbor Kernel: Transfers information from the most similar datasets based on meta-features like the number of instances and features.

Crucially, this approach maintains linear complexity ($O(T)$ per iteration). Unlike previous methods that rely on expensive ranking algorithms like SVM-Rank, this method only requires basic statistical recomputation, ensuring that the tuning process remains faster than the model training itself.

Experimental Evaluation

The method was evaluated using 42 real-world classification tasks from the LIBSVM repository and 210 synthetic datasets. The researchers tested the algorithm on tuning Logistic Regression and a complex ensemble of 30 parameters across seven different classifiers.

Key Results and Insights

The proposed method demonstrated significant improvements in both accuracy and efficiency:

  • Superior Accuracy: On synthetic datasets, the MKL-based transfer method consistently outperformed independent GPs and random search, particularly in "cold-start" scenarios (the first 5–10 trials).
  • Computational Scalability: The most striking result is the runtime efficiency. As the number of datasets increases, the CPU time for the proposed method grows linearly.
Computational efficiency comparison showing linear scaling of the proposed method versus the quadratic growth of previous state-of-the-art ranking methods.

The study also revealed that the "region of goodness"—the area of the hyperparameter space where models perform well—is often consistent across similar tasks. By exploiting this "common geography" through simple z-score normalization, the method achieves results comparable to much more complex models.

Conclusion

This research demonstrates that simplicity can be a powerful driver of scalability in hyperparameter optimization. By standardizing response surfaces and using a dual-kernel approach, the proposed method enables effective transfer learning without the computational overhead of ranking-based surrogates. Future work will focus on automating the selection of dataset meta-features and kernel weights to further refine the transfer process.

Summary updated

Abstract

We propose a fast and effective algorithm for automatic hyperparameter tuning that can generalize across datasets. Our method is an instance of sequential model-based optimization (SMBO) that trans...
BibTeX
@inproceedings{yogatama2014efficient,
  author = {Dani Yogatama and Gideon Mann},
  title = {{Efficient Transfer Learning Method for Automatic Hyperparameter Tuning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
  url = {https://openreview.net/forum?id=7CwUvzmDnb},
}
Figure