Introduction
Traditional word embedding models, such as the original Skip-Gram, operate under a "one-vector-per-word" assumption. This approach is fundamentally limited when dealing with polysemy—words that possess multiple distinct meanings. For example, the word "bank" refers to a financial institution in one context and a geographical feature in another. Forcing these disparate meanings into a single vector creates an "average" representation that fails to accurately capture any specific sense, ultimately degrading the performance of downstream natural language processing tasks.
The NTSG Architecture
To address this limitation, the Neural Tensor Skip-Gram (NTSG) model introduces a framework that integrates latent topics to distinguish between word senses. Unlike previous multi-prototype models that relied on local context clustering, NTSG leverages global topics assigned via Latent Dirichlet Allocation (LDA) to provide a consistent view of word senses across an entire corpus.
As illustrated in Figure 1, NTSG extends the standard Skip-Gram architecture by replacing simple linear layers with a tensor layer. While standard models use a bilinear interaction (dot product) between a word and its context, NTSG explicitly models the ternary (three-way) relationship between the target word ($w$), its assigned topic ($t$), and the context word ($c$).
To manage the computational complexity and potential overfitting associated with high-dimensional tensors, the authors utilize low-rank tensor factorization. By factorizing the tensor into low-rank matrices, the model's complexity is reduced from $O(d^2k)$ to $O(rdk)$, ensuring the architecture remains both expressive and computationally feasible.
Evaluation and Key Results
The NTSG model was trained on a large-scale Wikipedia corpus and evaluated across word similarity and text classification tasks:
- Contextual Word Similarity: On the Stanford Contextual Word Similarities (SCWS) dataset, the NTSG-2 variant achieved a Spearman correlation of 69.5%, surpassing the previous state-of-the-art multi-prototype models.
- Text Classification: In the 20NewsGroup task, NTSG significantly outperformed the standard Skip-Gram baseline, achieving an F1-score of 82.4% compared to 74.2%.
Beyond quantitative metrics, the model demonstrated remarkable qualitative success in sense disambiguation. By conditioning word representations on latent topics, the model can identify distinct nearest neighbors for different meanings of the same word.
As shown in Figure 6, the model successfully separates the various senses of words like "apple" (distinguishing the technology company from the fruit) and "fox" (distinguishing television networks from the animal). This ability to isolate specific meanings allows for more nuanced and accurate text representations.
Conclusion and Future Work
The NTSG model proves that capturing ternary interactions between words, topics, and contexts is vital for handling polysemy. While the current model uses a two-step process involving LDA, future research may explore unified architectures that learn latent topics and word embeddings simultaneously, as well as more sophisticated methods for composing word and topic vectors.
Summary updated