Causally Denoise Word Embeddings Using Half-Sibling Regression

Zekun Yang, Tianlin Liu

Abstract

Distributional representations of words, or word vectors, are fundamental to modern natural language processing. However, these vectors often contain confounding noise from the training corpus that limits their performance. We introduce a novel postprocessing scheme based on Half-Sibling Regression (HSR), a causal inference framework. By identifying and subtracting shared noise between content words and function words, our method provides an interpretable and data-efficient way to enhance off-the-shelf embeddings. Evaluated on lexical-level and downstream tasks, HSR reaches state-of-the-art performance across Word2Vec, GloVe, and Paragram embeddings.

Introduction

Word embeddings like Word2Vec and GloVe serve as the bedrock for most NLP applications. Despite their utility, these vectors are frequently "noisy" and lack isotropy—meaning they are not uniformly distributed in space and are often biased by word frequency. Existing postprocessing solutions typically fall into two categories: lexical approaches that rely on external dictionaries (like WordNet) and spatial approaches that use geometric heuristics (like PCA-based subtraction). While effective, lexical methods are limited by dictionary coverage, and spatial methods often lack theoretical interpretability.

Methodology: The Causal Logic of HSR

We propose adapting Half-Sibling Regression (HSR) to the NLP domain to provide a theoretically grounded denoising method. The core intuition relies on the relationship between content words (e.g., "apple") and function words (e.g., "the").

The causal graph for Half-Sibling Regression, where Q is the true semantic signal, N is the shared noise, and X and Y are observed vectors.

As shown in the causal graph above, we treat a content word $Y$ and a function word $X$ as "half-siblings." Both share a common parent $N$, representing the confounding noise inherent in the training corpus or algorithm. However, only the content word $Y$ possesses the parent $Q$, representing its true semantic meaning. Because function words are semantically thin, any information they share with content words is assumed to be the shared noise $N$.

The Algorithm

  1. Noise Identification: We use Ridge Regression to predict the content word vectors ($Y$) using a set of function word vectors ($X$) as features. This prediction serves as an estimate of the shared noise $E[Y|X]$.
  2. Denoising: The "clean" semantic signal $\hat{Q}$ is obtained by subtracting the predicted noise from the original vector: $\hat{Q} = Y - E[Y|X]$.
  3. Symmetry: To ensure the entire embedding space is consistent, we also denoise function words using a small sample of content words as predictors.

Experimental Results

We evaluated HSR-denoised embeddings (HSR-RR) against several baselines, including original vectors and state-of-the-art methods like All-but-the-Top (ABTT) and Conceptor Negation (CN).

Semantic Textual Similarity (STS)

The most significant gains were observed in STS tasks. Across 20 different benchmarks from SemEval (2012–2015), HSR-RR consistently outperformed competitors.

Performance comparison on Semantic Textual Similarity (STS) tasks across Word2Vec, GloVe, and Paragram embeddings.

As detailed in the table above, HSR-RR achieved average improvements of 4.71% to 7.54% over previous best results. The improvement over raw embeddings was even more pronounced, reaching up to 22.06% for GloVe vectors. This suggests that the causal framework is highly effective at isolating and removing the specific noise components that hinder similarity measurements.

Word Similarity and Sentiment Analysis

On the SimVerb-3500 task, which requires distinguishing true semantic similarity from simple association, HSR-RR achieved the top performance across all embedding types. Furthermore, in downstream sentiment analysis tasks (e.g., IMDB, Amazon Reviews), HSR-RR reached the highest classification accuracy, particularly when applied to Paragram embeddings.

Conclusion

Half-Sibling Regression offers a powerful, interpretable, and computationally efficient framework for word embedding postprocessing. By leveraging stop words as proxies for corpus-level noise, we can significantly improve the quality of static embeddings without the need for external linguistic resources or expensive retraining. Future work will explore applying these causal denoising principles to contextualized embeddings like BERT and GPT.

Summary updated

Abstract

Distributional representations of words, also known as word vectors, have become crucial for modern natural language processing tasks due to their wide applications. Recently, a growing body of word vector postprocessing algorithm has emerged, aiming to render off-the-shelf word vectors even stronger. In line with these investigations, we introduce a novel word vector postprocessing scheme under a causal inference framework. Concretely, the postprocessing pipeline is realized by Half-Sibling Regression (HSR), which allows us to identify and remove confounding noise contained in word vectors. Compared to previous work, our proposed method has the advantages of interpretability and transparency due to its causal inference grounding. Evaluated on a battery of standard lexical-level evaluation tasks and downstream sentiment analysis tasks, our method reaches state-of-the-art performance.
BibTeX
@inproceedings{yang2020causally,
  author = {Zekun Yang and Tianlin Liu},
  title = {{Causally Denoise Word Embeddings Using Half-Sibling Regression}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=FSCjNQmDOYY},
}
Figure