Causally Denoising Word Embeddings via Half-Sibling Regression
Abstract
Word embeddings trained on large corpora often inherit systematic noise, such as frequency-based biases, which can degrade performance on semantic tasks. This paper introduces a novel denoising framework based on Half-Sibling Regression (HSR). By identifying function words as semantically-null "half-siblings" that share the same corpus-induced noise as content words, the proposed HSR-RR method effectively isolates and removes confounding signals. Experimental results across Word2Vec, GloVe, and Paragram embeddings demonstrate that HSR-RR achieves state-of-the-art performance on Semantic Textual Similarity (STS) tasks and significant improvements in word similarity benchmarks without requiring external linguistic resources.
Introduction
Word embeddings like Word2Vec and GloVe serve as the foundation for many modern Natural Language Processing (NLP) applications. However, these vectors are often contaminated by noise inherent to the training data. Existing denoising methods typically fall into two categories: lexical approaches, which rely on external knowledge bases like WordNet, and spatial approaches, which use heuristic transformations such as the "All-But-The-Top" (ABTT) method to remove the first few principal components. While effective, these methods often lack theoretical grounding or generalizability to out-of-vocabulary words. This work proposes a causally-motivated alternative that is both interpretable and computationally efficient.
Methodology
The core of the proposed approach is the application of Half-Sibling Regression (HSR) to word vectors. In a causal framework, two variables are considered half-siblings if they share a common parent. Here, content words ($Y$) and function words ($X$) are treated as half-siblings because they are both influenced by the same corpus-level noise ($N$).
As shown in the causal graph, $Y$ contains a clean semantic signal ($Q$) and noise ($N$), while $X$ contains primarily noise ($N$) with minimal semantic signal. By regressing the content word vectors onto the function word vectors, the model learns to predict the shared noise component. Subtracting this prediction from the original embedding recovers a denoised signal. The authors primarily employ Ridge Regression (HSR-RR) for this task due to its simplicity and efficiency.
Experimental Evaluation
The authors evaluated HSR-RR using three major pre-trained embedding models—Word2Vec, GloVe, and Paragram—across a wide range of benchmark tasks:
- Word Similarity: Evaluated on 7 datasets, including WordSim-353, SimLex-999, and SimVerb-3500.
- Semantic Textual Similarity (STS): 20 tasks from SemEval (2012–2015) and the SICK dataset.
- Downstream Tasks: Sentiment analysis on standard datasets including IMDB and Amazon reviews.
Results and Discussion
HSR-RR demonstrates superior performance across nearly all benchmarks. Notably, on 20 STS tasks, HSR-RR outperformed existing baselines like ABTT and CN by significant margins.
The empirical success of HSR-RR is particularly evident in the STS tasks, where it improved GloVe embeddings by over 22% compared to raw vectors. The results indicate that function words serve as excellent proxies for corpus noise. Unlike heuristic methods, HSR provides a clear causal justification for the information being removed. Furthermore, the method is "data-thrifty," requiring only a small set of common stop words rather than massive external databases. On the challenging SimVerb-3500 dataset, HSR-RR consistently achieved the highest scores, highlighting its ability to preserve fine-grained semantic nuances that are often lost in other denoising processes.
Conclusion
This paper presents a robust, causally-motivated method for denoising word embeddings. By leveraging the shared noise between content and function words, HSR-RR provides a theoretically sound and empirically successful alternative to existing heuristic methods. Future research may extend this approach to contextualized embeddings from models like BERT or explore more complex regression models to capture inter-word correlations.
Summary updated