Denoising Semantic Embeddings via Common Component Removal

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2102 papers139 conceptsfrozen 2026-10-04

When we project words or sentences into high-dimensional vector spaces, we expect them to be distributed according to their meaning. However, empirical evidence across nearly a decade of research shows that these spaces are surprisingly "lumpy." Instead of filling the space uniformly, embeddings often cluster into a narrow cone—a phenomenon known as anisotropy—dominated by a few "rogue" dimensions that encode frequency, syntax, and common linguistic noise rather than specific semantic content.

A powerful and remarkably simple family of techniques has emerged to fix this: identifying the most dominant directions of variance (usually via Principal Component Analysis) and simply subtracting them.

The Geometry of Common Discourse

The core intuition behind this research direction was popularized by Sanjeev Arora, Yingyu Liang, and Tengyu Ma in their seminal work on Smooth Inverse Frequency (SIF). They argued that any given sentence embedding is composed of two parts: the actual semantic signal and a "common discourse vector" that represents the shared variance across the entire corpus.

Think of this common component as the "background hum" of language—the filler words, grammatical structures, and high-frequency patterns that appear in almost every sentence regardless of its specific meaning. By applying Principal Component Analysis (PCA) to a set of sentence vectors and subtracting the projection of each vector onto the first principal component, we effectively "mute" this background hum. This denoising step reveals the unique semantic signal, allowing models to better distinguish between subtle differences in meaning.

All but the Top

While SIF focused on sentences, Jiaqi Mu and Pramod Viswanath extended this logic to word representations in their All-but-the-Top approach. They observed that standard embeddings like word2vec and GloVe suffer from a "frequency bias": the most dominant dimensions in the vector space often correspond to how often a word appears, not what it means.

By removing the top $D$ principal components (where $D$ is typically small, like 1 to 5), they achieved consistent performance boosts across a wide range of NLP tasks. This process makes the embedding space more isotropic—meaning the vectors are spread out more evenly in all directions. When the space is isotropic, cosine similarity becomes a much more reliable measure of semantic relatedness because it is no longer "hijacked" by the massive shared variance of the top components.

Rogue Dimensions in Transformers

The transition from static word vectors to contextualized embeddings like BERT didn't solve the problem; it arguably made it more visible. William Timkey and Marten van Schijndel identified rogue dimensions in Transformer models—a tiny handful of dimensions that disproportionately dominate the vector space. These outliers create a "geometric illusion" of poor representational quality.

Similarly, Richong Zhang and colleagues found that the first principal component in LLM embedding spaces often represents high-frequency, low-information tokens. Adjusting or removing this component acts as a high-pass filter, clarifying the relationship between the embedding and the most meaningful keywords in the input.

Solving the Hubness Problem

One of the most practical applications of this technique is in unsupervised bilingual dictionary induction. When trying to align two different languages (like English and Japanese) without a parallel corpus, researchers often run into the "hubness" problem: a few words in the vector space act as the nearest neighbors to almost every other word.

Yanyang Li and others demonstrated that discarding the highest-variance principal component significantly reduces hubness. By removing the global bias components that represent noise rather than semantic structure, the alignment algorithms can focus on the stable, high-dimensional features that actually correspond across languages.

Beyond Text: Speech and Diffusion

The principle of common component removal is now being applied far beyond traditional NLP:

  • Speech: In the Eta-WavLM framework, Giuseppe Ruggiero and colleagues use PCA to isolate and subtract speaker identity from speech representations. This allows them to strip away "who" is talking to better capture "what" is being said, improving tasks like voice conversion.
  • Generative Models: Trung X. Pham and Kang Zhang discovered a semantic bottleneck in Diffusion Transformers. They found that by pruning the low-magnitude "tail" dimensions of conditional embeddings, they could reduce semantic interference and actually improve image generation quality.
  • Clinical Diagnostics: Akshay Budhkar and Frank Rudzicz used this technique to detect Alzheimer’s disease from speech transcripts. Removing the first principal component helped isolate diagnostic signals from common linguistic noise, leading to state-of-the-art classification accuracy.

Why It Works: The Signal-to-Noise Filter

The recurring theme across these 88 papers is that high-dimensional embeddings are not "pure" representations. They are mixtures of specific semantic signals and broad, non-informative patterns.

Whether it is called "common component removal," "denoising," or "whitening," the mechanism is the same: treat the most dominant directions of variance as noise. By projecting our data into the subspace that remains, we force our models to stop relying on easy, high-frequency patterns and instead utilize the nuanced, discriminative features that define true meaning.

Papers in this version

Showing 60 of 102 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 58 papers
  2. Version 1 · 2026-10-04 · 102 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_common_component_removal_db27e56e,
  title = {common component removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/common-component-removal/rdr_db27e56e55314dada913129ae0c1dc0b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_denoising_semantic_embeddings_via_common_db27e56e_v1,
  title = {Denoising Semantic Embeddings via Common Component Removal},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/denoising-semantic-embeddings-via-common-component-removal/rdr_db27e56e55314dada913129ae0c1dc0b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}