Abstract
When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts. They can have a big effect on performance, but they are often uninteresting to researchers or practitioners who simply need a module that performs well. We apply sequential model-based optimization over this space of choices and show that it makes standard linear models competitive with more sophisticated, expensive state-of-the-art methods based on latent variables or neural networks on various topic classification and sentiment analysis problems. Our approach is a first step towards black-box NLP systems that work with raw text and do not require manual tuning.
Introduction
In Natural Language Processing (NLP), a significant amount of effort is often dedicated to "nuisance" decisions regarding text representation. Practitioners must decide on n-gram ranges, weighting schemes (e.g., tf-idf vs. binary), and preprocessing steps like stopword removal. These choices are typically made through heuristic rules or exhaustive trial-and-error, yet they profoundly impact model performance. Furthermore, a representation optimized for one domain, such as movie reviews, may not generalize to another, such as congressional transcripts. This paper proposes automating these decisions to create "black-box" NLP systems that find optimal configurations directly from raw text.
Methods
The researchers utilize Sequential Model-Based Optimization (SMBO), specifically the Tree-structured Parzen Estimator (TPE), to navigate the representational search space. The underlying classifier is fixed as a standard Logistic Regression (LR) model to demonstrate that performance gains come from representational tuning rather than model complexity.
The optimization covers several dimensions, including n-gram range (up to trigrams), weighting schemes (tf, tf-idf, or binary), stopword removal, and regularization parameters ($L_1$ vs $L_2$ and their strengths). The TPE algorithm uses an acquisition function (Expected Improvement) to balance exploration and exploitation, aiming to find near-optimal configurations within a limited budget of only 30 trials per dataset.
Results
The approach was evaluated across diverse benchmarks, including the Stanford Sentiment Treebank (SST), Amazon Electronics, IMDB, and Congressional Vote datasets. A key finding is that there is no universal "best" representation; optimal settings vary significantly across tasks. For instance, the "Congress vote" task favored bigrams, while "Amazon electronics" benefited from a mix of unigrams, bigrams, and trigrams.
Crucially, the tuned LR models proved highly competitive with more complex architectures. On the SST dataset, the automated search achieved 82.4% accuracy, matching Recursive Neural Networks and outperforming standard SVM baselines.
Similar results were observed in topic classification (20 Newsgroups), where the method reached 87.84% accuracy, significantly outperforming Discriminative RBMs (76.2%). In almost all cases, the automated search outperformed expert-selected baselines from previous literature.
Conclusion
This work demonstrates that text representation is a critical hyperparameter that can be successfully optimized jointly with model parameters. By treating representation as a search problem, simple linear models can achieve state-of-the-art performance with minimal manual intervention. Future work includes expanding the search space to include modern embeddings, exploring meta-learning to transfer optimization knowledge across datasets, and applying the framework to unsupervised tasks.
Summary updated