Abstract
Short text classification (STC) faces significant hurdles due to data sparseness and the high frequency of unseen words in test data. Traditional methods relying on discrete word counts often fail to capture semantic nuances in limited contexts. This paper introduces a robust framework that represents text categories as Gaussian distributions within a continuous word embedding space. By leveraging embeddings trained on a massive unlabeled corpus (Wikipedia), the model effectively "pre-learns" semantic relationships, allowing it to generalize to unseen vocabulary. Experimental results on a benchmark web snippets dataset demonstrate that the proposed method achieves an accuracy of 85.48%, significantly outperforming traditional TF-IDF and topic-modeling approaches.
Introduction
Categorizing short snippets of text—such as tweets, search results, or instant messages—is a notoriously difficult task in natural language processing. Unlike long-form documents, short texts suffer from extreme data sparseness, containing too few words to establish reliable statistical patterns using traditional methods like TF-IDF or Bag-of-Words.
Furthermore, supervised learning for STC typically requires massive amounts of manually labeled data. Even with such data, models often fail when encountering "unseen words"—terms present in the test set but missing from the training data. This gap in vocabulary is a primary driver of performance degradation in real-world applications.
The Challenge of Unseen Vocabulary
A critical observation in this study is the magnitude of the "unseen word" problem. In the benchmark Web Snippets dataset, a substantial portion of the test vocabulary is entirely absent from the training set. This phenomenon persists even after applying standard preprocessing techniques like stemming.
As shown in the table above, over 43% of the words in the test set were new to the model. Traditional discrete models, which rely on exact matches, are fundamentally unequipped to handle such a high rate of novel terminology.
Methodology: Gaussian-Embedding Framework
To bridge this context gap, the authors propose a three-step approach that moves classification from a discrete space to a continuous semantic space:
- Distributed Representations: Using Word2Vec, word embeddings are trained on the entire English Wikipedia (approx. 14.9 million articles). This maps every word into a high-dimensional vector space where semantically similar words (e.g., "physician" and "doctor") are geometrically close.
- Gaussian Modeling: Instead of simple word counts, each text category (or "theme") is treated as a Gaussian distribution in the embedding space. The model estimates the mean ($\mu$) and covariance ($\Sigma$) for each category based on the word vectors found in the training data.
- Bayesian Classification: To classify a new snippet, the model calculates the probability that the document’s word vectors were generated by each category's Gaussian distribution. Using Bayesian rules, it assigns the text to the most probable theme.
This "semantic smoothing" allows the model to correctly classify a snippet containing "physician" even if that specific word never appeared in the training labels for the "Health" category, provided it has learned the relationship between "physician" and other health-related terms during the unsupervised embedding phase.
Key Results
The proposed Gaussian-embedding approach was evaluated against traditional baselines on the Web Snippets dataset, which includes 12,340 snippets across 8 domains such as business, health, and sports.
As illustrated in the figure, the Proposed Method consistently outperforms both the standard MaxEnt + TF-IDF (Method 1) and the topic-model-enhanced MaxEnt + LDA (Method 2). While traditional methods struggle with smaller training sets, the proposed model maintains high performance throughout, reaching a peak accuracy of 85.48% with 400-dimension embeddings. Notably, the figure also tracks the percentage of unseen words (Original and After Stemming), which decreases as training data grows but remains a significant hurdle that the proposed method successfully navigates.
Conclusion
By representing categories as distributions in a continuous vector space, this research provides a powerful solution to the data sparseness and unseen word problems in short text classification. The ability to leverage massive unlabeled corpora to "pre-learn" language semantics significantly reduces the burden on labeled training data. Future work will explore applying this distributional approach to more complex text genres and investigating non-Gaussian distributions to better capture the nuances of diverse topics.
Summary updated