Introduction
Short text snippets, such as sentences or news headers, pose a significant challenge for traditional Natural Language Processing (NLP) techniques. Standard vector representations, like Bag-of-Words (BoW) or Explicit Semantic Analysis (ESA), rely on statistical overlap between documents. While effective for long-form text, these representations become extremely sparse for short documents. Two semantically related snippets may share no common words or Wikipedia concepts, resulting in a cosine similarity of zero. This "sparsity trap" severely degrades performance in tasks like document classification and semantic similarity measurement.
The Dense-ESA Framework
To address this, we introduce Dense-ESA, an unsupervised framework that densifies sparse vector representations by leveraging the relational information in dense word embeddings (e.g., word2vec). Unlike standard ESA, which only considers exact matches between concept dimensions, Dense-ESA "aligns" non-zero dimensions based on their semantic proximity.
The framework utilizes word2vec embeddings trained on Wikipedia to represent each ESA concept title as a dense 200-dimensional vector. We explore three distinct alignment mechanisms:
- Similarity Augmentation ($S_A$): A many-to-many approach averaging all pairwise similarities between terms, dampened by an RBF kernel.
- Maximum Matching ($S_M$): A many-to-one approach where each term in the source text is matched to its most similar counterpart in the target.
- Hungarian Method ($S_H$): A one-to-one approach using combinatorial optimization to find the globally optimal alignment between unique terms.
Experimental Results
The proposed methods were evaluated across three benchmarks: the 20-Newsgroups dataset for dataless classification, the Lee et al. (2005) document similarity set, and the ACE2005 event classification task.
Dataless Classification
In the "dataless" setting, documents are classified by comparing their vector representations directly to label vectors (e.g., the concept "Autos"). Dense-ESA significantly outperforms standard ESA, particularly on "difficult" pairs like Autos vs. Motorcycles, where vocabulary overlap is minimal.
As shown in the box plots above, Dense-ESA provides a clearer separation of similarity scores compared to the baseline ESA, especially in the challenging Autos vs. Motorcycles task where standard ESA achieved only 46.64% accuracy compared to Dense-ESA's 59.65%.
Impact of Sparsity
A key finding is that Dense-ESA maintains high accuracy even as the number of available concepts decreases (increasing sparsity).
In the performance plots above, we observe that while standard ESA (blue line) requires a large number of concepts to achieve peak performance, Dense-ESA variants (particularly the "Average" and "Hungarian" versions) reach high accuracy much earlier. This demonstrates the robustness of densification in low-information scenarios.
Conclusion
Dense-ESA effectively bridges the gap between interpretable sparse representations and relational dense embeddings. By aligning concepts rather than relying on exact matches, the method significantly improves short-text similarity measurement without the need for labeled training data. Future work will explore the computational optimization of the Hungarian method and the integration of these densified features into supervised deep learning models.
Summary updated