Dheeraj Mekala, Vivek Gupta, Bhargavi Paranjape, Harish Karnick. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Abstract
This paper introduces SCDV (Sparse Composite Document Vectors), a novel method for representing documents that bridges the gap between simple word embeddings and complex topic models. By combining soft clustering with sparsity constraints, the authors achieve state-of-the-art performance in text classification while significantly reducing computational overhead.
Introduction
In Natural Language Processing (NLP), representing a whole document as a mathematical vector is a fundamental challenge. Early methods like Bag-of-Words (BoW) ignore semantics, while modern word embeddings (like Word2Vec) are effective for individual words but difficult to aggregate into meaningful document representations.
The authors identify three specific flaws in existing approaches:
- The Polysemy Problem: Most models assume a word has a single meaning. However, a word like "apple" requires different representations in a culinary context versus a technological one.
- The "Same Space" Fallacy: Many models embed documents in the same low-dimensional space as words. This limits expressive power; a complex document shouldn't necessarily be confined to the same 300-dimensional space as a single word.
- Computational Efficiency: High-performing models often rely on Latent Dirichlet Allocation (LDA) or complex neural architectures that are slow to train and expensive to store.
Methodology
SCDV addresses these issues through a three-step pipeline designed to capture semantic nuance while maintaining efficiency.
1. Soft Clustering with GMMs
Instead of assigning a word to just one topic (hard clustering), SCDV uses Gaussian Mixture Models (GMM) to assign words to multiple clusters with varying probabilities. This captures the different "senses" or contexts of a word. For example, the model might learn that "bank" belongs 60% to a "finance" cluster and 40% to a "geography" cluster.
2. Document Topic-Vector Formation
For every word in a document, the model creates a "word-topic vector" by weighting the word's embedding by its cluster probabilities and its Inverse Document Frequency (IDF). These are concatenated into a high-dimensional vector ($K$ clusters $\times$ $d$ embedding dimensions) and summed to represent the document.
3. Sparsification
The resulting vectors are dense and high-dimensional. However, the authors observed that most values in these vectors are near zero. They apply a thresholding technique to zero out these values, creating a Sparse vector. This reduces noise and makes downstream machine learning tasks (like SVM classification) significantly faster.
Key Results
The authors evaluated SCDV on several standard benchmarks, including 20NewsGroup, Reuters-21578, and the TREC Collection.
- Classification Accuracy: On the 20NewsGroup dataset, SCDV achieved 84.6% accuracy, outperforming the previous state-of-the-art NTSG (82.6%) and significantly beating Doc2Vec (72.4%).
- Efficiency and Storage: SCDV is remarkably efficient. Compared to the Bag of Words Vector (BoWV) model, SCDV reduced storage requirements by 80% (from 1.1 GB to 236 MB). Document vector formation was roughly 4x faster than TWE-1 and 8x faster than BoWV.
- Topic Coherence: Qualitative analysis showed that SCDV produces more semantically coherent topics than traditional LDA, as measured by topic coherence scores.
Discussion and Future Work
SCDV demonstrates that high dimensionality combined with sparsity can overcome the "curse of dimensionality." While a $K \times d$ vector (e.g., 12,000 dimensions) sounds massive, the sparsity ensures that only relevant features are kept, acting as a structural regularizer that prevents overfitting.
While powerful, the model is sensitive to hyperparameters like the number of clusters and the sparsity threshold. Furthermore, as a "bag-of-words" style aggregation, it still largely ignores word order (syntax). Future research may explore integrating SCDV into deep learning pipelines as a pre-trained feature extractor or utilizing it in real-time information retrieval systems where speed is critical.
Summary updated