Abstract
Short texts usually encounter the problem of data sparseness, as they do not provide sufficient term co-occurrence information. In this paper, we show how to mitigate the problem in short text classification through word embeddings. We assume that a short text document is a specific sample of one distribution in a Gaussian-Bayesian framework. Furthermore, a fast clustering algorithm is utilized to expand and enrich the context of short text in embedding space. This approach is compared with those based on the classical bag-of-words approaches and neural network based methods. Experimental results validate the effectiveness of the proposed method.
Introduction
Short texts—such as tweets, instant messages, and product snippets—are notoriously difficult for machines to classify. The primary culprit is data sparseness: because these documents are so brief, they lack sufficient context or "term co-occurrence."
Traditional methods like Bag-of-Words (BoW) rely on overlapping words to find similarities. If one text says "I love my phone" and another says "This mobile is great," a BoW model might fail to see the connection because "phone" and "mobile" are different strings. While deep learning has helped, many neural networks require massive amounts of data to generalize well, which isn't always available or computationally cheap. This paper introduces a Gaussian-Bayesian framework that overcomes these limitations by leveraging word embeddings and unsupervised expansion.
Methodology
The proposed framework classifies short texts through a three-step pipeline:
1. Word Representation
The system utilizes Word2Vec (Skip-gram or CBOW) trained on a massive Wikipedia dump (October 2014, ~15 million articles) to map words into a continuous vector space. This ensures that semantically similar words (like "phone" and "mobile") are positioned close to each other, addressing the vocabulary mismatch problem inherent in BoW models.
2. Short Text Expansion
To mitigate brevity, a fast clustering algorithm based on "density peaks searching" identifies "semantic cliques"—groups of highly related words. For every word in a short text, the system identifies its nearest neighbors in the embedding space and adds them to the text, providing necessary context without requiring manually curated ontologies like WordNet.
3. Gaussian-Bayesian Classification
Instead of a standard neural network, the model treats each classification theme (e.g., "Sports" or "Business") as a Gaussian distribution of word vectors. During training, the mean and covariance for each theme are estimated. For a new short text, the Bayesian rule calculates which theme’s distribution most likely generated those specific word vectors. This probabilistic approach is more transparent and computationally efficient than "black box" deep learning architectures.
Experimental Results
The model was evaluated using the Web Snippets dataset, which contains 12,340 snippets across 8 domains (Business, Computers, Health, etc.), with an average length of just 18 words.
As shown in the results table, the Gaussian-Bayesian approach (85.8% with expansion) outperformed state-of-the-art deep learning models, including CCNN (85.5%) and Semantic-CNN (85.1%). It significantly outperformed traditional BoW methods and simpler neural architectures like LSTMs (63.0%).
Analysis of the training process shows that the model demonstrates high resilience to the "unseen words" problem. As training data increases, the percentage of unseen words in the test set decreases, and the accuracy of the proposed method stabilizes at a high level, consistently outperforming LDA and TF*IDF based baselines even with limited training samples.
Conclusion
The Gaussian-Bayesian framework provides a robust solution for short text classification by leveraging the distributional properties of word embeddings. By combining unsupervised text expansion with a probabilistic classification model, the approach achieves state-of-the-art performance while remaining computationally efficient. Future work will explore the model's performance on noisier datasets, such as microblogs containing slang and typos, and investigate methods to scale covariance matrix estimation for higher-dimensional spaces.
Summary updated