Abstract
Word embedding models like Word2Vec and GloVe typically require massive datasets to produce high-quality representations. In low-data regimes, these models often overfit, resulting in brittle vectors that fail to generalize, particularly for rare words. This paper presents Robust Gram (RG), a regularized embedding approach that enforces a similarity constraint between target and context vectors. By penalizing the Frobenius norm of the difference between these two embedding spaces, RG reduces model complexity and prevents overfitting. Experimental results demonstrate that RG consistently outperforms standard Skip-Gram and GloVe models on word similarity benchmarks, showing remarkable robustness in the presence of rare words and limited training data.
Introduction
Modern Natural Language Processing (NLP) relies heavily on word embeddings to capture semantic relationships. However, popular methods like Skip-Gram (SG) and GloVe are data-hungry. When trained on small or noisy corpora, these models tend to memorize noise rather than learn generalizable patterns. A primary cause of this instability is the standard practice of maintaining separate embedding matrices for words acting as "targets" and words acting as "contexts," which effectively doubles the number of parameters the model must learn.
In "Robust Gram Embeddings" (EMNLP 2016), Kekeç and Tax propose a principled mathematical solution: enforcing symmetry between the target and context spaces. This approach, termed Robust Gram (RG), acts as a powerful prior that guides the model toward more robust features in data-scarce environments.
Methodology
The core innovation of Robust Gram lies in its modified objective function. While standard models minimize a loss based on word co-occurrences, RG introduces two regularization terms:
- Weight Decay ($g_1$): A standard $\ell_2$ regularization to prevent vector values from exploding.
- Similarity Constraint ($g_2$): A penalty on the distance between the target embedding matrix ($\Psi$) and the context embedding matrix ($\Phi$), calculated using the Frobenius norm.
The final objective function is defined as: $$LL + \lambda_1 (\text{Weight Decay}) + \lambda_2 ||\Psi - \Phi||^2_F$$
By increasing $\lambda_2$, the model is forced to find a shared representation that serves both target and context roles effectively. This reduction in the "degrees of freedom" prevents the model from overfitting to the specific quirks of small datasets.
Experimental Results
The authors evaluated RG against several baselines, including Skip-Gram (SG), CBoW, and GloVe, using a 15-million-word subset of Wikipedia. Performance was measured using Spearman’s $\rho$ correlation on various word similarity benchmarks.
Performance Across Benchmarks
As shown in the table below, RG matches or exceeds the performance of baseline models across nearly all tasks. Notably, it achieves a significant advantage on the Rare Words (RW) dataset, which specifically targets infrequent terms that are usually difficult for standard models to represent accurately.
The Impact of Rare Words
A key insight from the study is RG's unique relationship with rare words. In standard embedding training, it is common practice to apply a "cut-off"—deleting words that appear fewer than $N$ times—to improve the quality of the remaining vectors. However, for RG, removing these words actually degrades performance.
As illustrated in Figure 1, the performance of RG (blue line) is highest when the cut-off is low, suggesting that the model is capable of extracting useful semantic signals from rare words that other models would treat as noise. This makes RG particularly valuable for specialized domains where data is naturally limited.
Conclusion
Robust Gram Embeddings offer a mathematically grounded approach to the data scarcity problem in NLP. By enforcing symmetry between target and context representations, the model achieves greater stability and better generalization than traditional unconstrained models. The success of RG on rare word tasks highlights the potential of regularization-based approaches to maximize the utility of limited training data. Future work may explore applying similar robustness constraints to dynamic, transformer-based architectures in low-resource settings.
Summary updated