Bayesian Optimization of Text Representations

Dani Yogatama, Lingpeng Kong, Noah A. Smith

Abstract

When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts. They can have a big effect on performance, but they are often uninteresting to researchers or practitioners who simply need a module that performs well. We apply sequential model-based optimization over this space of choices and show that it makes standard linear models competitive with more sophisticated, expensive state-of-the-art methods based on latent variables or neural networks on various topic classification and sentiment analysis problems. Our approach is a first step towards black-box NLP systems that work with raw text and do not require manual tuning.

Introduction

In Natural Language Processing (NLP), a significant amount of effort is often dedicated to "nuisance" decisions regarding text representation. Practitioners must decide on n-gram ranges, weighting schemes (e.g., tf-idf vs. binary), and preprocessing steps like stopword removal. These choices are typically made through heuristic rules or exhaustive trial-and-error, yet they profoundly impact model performance. Furthermore, a representation optimized for one domain, such as movie reviews, may not generalize to another, such as congressional transcripts. This paper proposes automating these decisions to create "black-box" NLP systems that find optimal configurations directly from raw text.

Methods

The researchers utilize Sequential Model-Based Optimization (SMBO), specifically the Tree-structured Parzen Estimator (TPE), to navigate the representational search space. The underlying classifier is fixed as a standard Logistic Regression (LR) model to demonstrate that performance gains come from representational tuning rather than model complexity.

Table 1: Hyperparameter search space for text representation and model parameters.

The optimization covers several dimensions, including n-gram range (up to trigrams), weighting schemes (tf, tf-idf, or binary), stopword removal, and regularization parameters ($L_1$ vs $L_2$ and their strengths). The TPE algorithm uses an acquisition function (Expected Improvement) to balance exploration and exploitation, aiming to find near-optimal configurations within a limited budget of only 30 trials per dataset.

Results

The approach was evaluated across diverse benchmarks, including the Stanford Sentiment Treebank (SST), Amazon Electronics, IMDB, and Congressional Vote datasets. A key finding is that there is no universal "best" representation; optimal settings vary significantly across tasks. For instance, the "Congress vote" task favored bigrams, while "Amazon electronics" benefited from a mix of unigrams, bigrams, and trigrams.

Crucially, the tuned LR models proved highly competitive with more complex architectures. On the SST dataset, the automated search achieved 82.4% accuracy, matching Recursive Neural Networks and outperforming standard SVM baselines.

Table 2: Comparison of accuracy on the Stanford Sentiment Treebank (SST). The tuned Logistic Regression (LR) matches state-of-the-art recursive models.

Similar results were observed in topic classification (20 Newsgroups), where the method reached 87.84% accuracy, significantly outperforming Discriminative RBMs (76.2%). In almost all cases, the automated search outperformed expert-selected baselines from previous literature.

Conclusion

This work demonstrates that text representation is a critical hyperparameter that can be successfully optimized jointly with model parameters. By treating representation as a search problem, simple linear models can achieve state-of-the-art performance with minimal manual intervention. Future work includes expanding the search space to include modern embeddings, exploring meta-learning to transfer optimization knowledge across datasets, and applying the framework to unsupervised tasks.

Summary updated

Abstract

When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts. They can have a big effect on performance, but they are often uninteresting to researchers or practitioners who simply need a module that performs well. We apply sequential model-based optimization over this space of choices and show that it makes standard linear models competitive with more sophisticated, expensive state-ofthe-art methods based on latent variables or neural networks on various topic classification and sentiment analysis problems. Our approach is a first step towards black-box NLP systems that work with raw text and do not require manual tuning.
BibTeX
@inproceedings{yogatama2015bayesian,
  author = {Dani Yogatama and Lingpeng Kong and Noah A. Smith},
  title = {{Bayesian Optimization of Text Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=B14SVzzubr},
}
Figure