Efficient Hyper-parameter Optimization for NLP Applications

Lidan Wang, Minwei Feng, Bowen Zhou, Bing Xiang, Sridhar MahadevanEMNLP 2015 · OpenReview

Abstract

Hyper-parameter optimization is an important problem in natural language processing (NLP) and machine learning. Recently, a group of studies has focused on using sequential Bayesian Optimization to solve this problem, which aims to reduce the number of iterations and trials required during the optimization process. In this paper, we explore this problem from a different angle, and propose a multi-stage hyper-parameter optimization that breaks the problem into multiple stages with increasingly amounts of data. Early stage provides fast estimates of good candidates which are used to initialize later stages for better performance and speed. We demonstrate the utility of this new algorithm by evaluating its speed and accuracy against state-of-the-art Bayesian Optimization algorithms on classification and prediction tasks.

Introduction

In machine learning and Natural Language Processing (NLP), the difference between a mediocre model and a state-of-the-art one often depends on the selection of hyper-parameters, such as learning rates, regularization constants, or the number of trees in a forest. While Bayesian Optimization (BO) has become a popular tool to automate this search by reducing the number of trials needed, it faces a major bottleneck in NLP: data scale. For modern NLP tasks with millions of examples, a single training run can take hours or days. Current hyper-parameter optimization (HPO) methods often focus on the dimensionality of the search space rather than the computational cost of the training data itself.

Multi-stage Bayesian Optimization

The proposed Multi-stage Bayesian Optimization algorithm breaks the optimization process into successive stages ($S$) with increasing amounts of training data.

  1. Early Stages: The algorithm uses a small, randomly sampled subset of the training data to perform sequential optimization. This quickly identifies "promising" regions of the hyper-parameter space at a low computational cost.
  2. Initialization (Warm-starting): The top $k$ configurations from the early stage are passed to the next stage.
  3. Later Stages: The algorithm moves to the full dataset, initializing its probabilistic surrogate model (using Gaussian Process priors and Expected Improvement) with the results from the previous stage.

This approach is orthogonal to other methods like parallelization and aims to eliminate waste by avoiding expensive iterations on the full dataset in regions that could be identified as suboptimal using only a fraction of the data.

Experimental Evaluation

The researchers evaluated the method on two distinct NLP tasks:

  • Yelp Dataset (Classification): Predicting star ratings (1–5) from 500,000 feature vectors using SVMs.
  • Commercial QA Dataset (Question Answering): Identifying correct answer candidates from 900,000 feature vectors using Boosted Regression Trees.

Results

The multi-stage approach was compared against standard BO on full data and BO on a small subset.

Precision@1 vs. Time for the QA dataset.

On the QA task, the multi-stage algorithm was 23% faster while reaching higher precision levels more quickly than standard BO. As shown in the precision vs. time plot, the multi-stage method (solid red line) significantly outperforms the baseline Bayesian optimization on full data (dotted green line) in the early stages of the search.

Classification accuracy vs. Time for the Yelp dataset.

On the Yelp classification task, the multi-stage approach achieved a 50% speedup per iteration. As shown in the accuracy vs. time plots, the multi-stage method approaches the performance ceiling much more rapidly than its competitors. Despite using less total compute time, it reached the same or slightly higher peak accuracy as standard full-data BO.

Conclusion

The multi-stage BO framework demonstrates that hyper-parameters performing well on data subsets are generally in the same neighborhood as those optimal for the full dataset. This allows for significant pruning of the search space at a low cost. Future work may explore optimal stage configurations, active sampling for subsets, and application to deep learning architectures.

Summary updated

Abstract

Hyper-parameter optimization is an important problem in natural language processing (NLP) and machine learning. Recently, a group of studies has focused on using sequential Bayesian Optimization to solve this problem, which aims to reduce the number of iterations and trials required during the optimization process. In this paper, we explore this problem from a different angle, and propose a multi-stage hyper-parameter optimization that breaks the problem into multiple stages with increasingly amounts of data. Early stage provides fast estimates of good candidates which are used to initialize later stages for better performance and speed. We demonstrate the utility of this new algorithm by evaluating its speed and accuracy against state-of-the-art Bayesian Optimization algorithms on classification and prediction tasks.
BibTeX
@inproceedings{wang2015efficient,
  author = {Lidan Wang and Minwei Feng and Bowen Zhou and Bing Xiang and Sridhar Mahadevan},
  title = {{Efficient Hyper-parameter Optimization for NLP Applications}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=HkE1tfz_ZS},
}
Figure