Abstract
Distributional representations of words, or word vectors, are fundamental to modern natural language processing. However, these vectors often contain confounding noise from the training corpus that limits their performance. We introduce a novel postprocessing scheme based on Half-Sibling Regression (HSR), a causal inference framework. By identifying and subtracting shared noise between content words and function words, our method provides an interpretable and data-efficient way to enhance off-the-shelf embeddings. Evaluated on lexical-level and downstream tasks, HSR reaches state-of-the-art performance across Word2Vec, GloVe, and Paragram embeddings.
Introduction
Word embeddings like Word2Vec and GloVe serve as the bedrock for most NLP applications. Despite their utility, these vectors are frequently "noisy" and lack isotropy—meaning they are not uniformly distributed in space and are often biased by word frequency. Existing postprocessing solutions typically fall into two categories: lexical approaches that rely on external dictionaries (like WordNet) and spatial approaches that use geometric heuristics (like PCA-based subtraction). While effective, lexical methods are limited by dictionary coverage, and spatial methods often lack theoretical interpretability.
Methodology: The Causal Logic of HSR
We propose adapting Half-Sibling Regression (HSR) to the NLP domain to provide a theoretically grounded denoising method. The core intuition relies on the relationship between content words (e.g., "apple") and function words (e.g., "the").
As shown in the causal graph above, we treat a content word $Y$ and a function word $X$ as "half-siblings." Both share a common parent $N$, representing the confounding noise inherent in the training corpus or algorithm. However, only the content word $Y$ possesses the parent $Q$, representing its true semantic meaning. Because function words are semantically thin, any information they share with content words is assumed to be the shared noise $N$.
The Algorithm
- Noise Identification: We use Ridge Regression to predict the content word vectors ($Y$) using a set of function word vectors ($X$) as features. This prediction serves as an estimate of the shared noise $E[Y|X]$.
- Denoising: The "clean" semantic signal $\hat{Q}$ is obtained by subtracting the predicted noise from the original vector: $\hat{Q} = Y - E[Y|X]$.
- Symmetry: To ensure the entire embedding space is consistent, we also denoise function words using a small sample of content words as predictors.
Experimental Results
We evaluated HSR-denoised embeddings (HSR-RR) against several baselines, including original vectors and state-of-the-art methods like All-but-the-Top (ABTT) and Conceptor Negation (CN).
Semantic Textual Similarity (STS)
The most significant gains were observed in STS tasks. Across 20 different benchmarks from SemEval (2012–2015), HSR-RR consistently outperformed competitors.
As detailed in the table above, HSR-RR achieved average improvements of 4.71% to 7.54% over previous best results. The improvement over raw embeddings was even more pronounced, reaching up to 22.06% for GloVe vectors. This suggests that the causal framework is highly effective at isolating and removing the specific noise components that hinder similarity measurements.
Word Similarity and Sentiment Analysis
On the SimVerb-3500 task, which requires distinguishing true semantic similarity from simple association, HSR-RR achieved the top performance across all embedding types. Furthermore, in downstream sentiment analysis tasks (e.g., IMDB, Amazon Reviews), HSR-RR reached the highest classification accuracy, particularly when applied to Paragram embeddings.
Conclusion
Half-Sibling Regression offers a powerful, interpretable, and computationally efficient framework for word embedding postprocessing. By leveraging stop words as proxies for corpus-level noise, we can significantly improve the quality of static embeddings without the need for external linguistic resources or expensive retraining. Future work will explore applying these causal denoising principles to contextualized embeddings like BERT and GPT.
Summary updated