Centering Embedding Spaces for Vector Alignment

16 concepts12 papers

When we train word or sentence representations across different languages, time periods, or domains, the underlying semantic structures are often remarkably similar. A dog relates to a cat in English much like it does in French, and a medical term carries similar contextual relationships whether it appears in a general corpus or a specialized textbook.

Yet, if you plot these embedding spaces within the same coordinate system, they rarely overlap. Instead, they cluster into isolated neighborhoods. This happens because models inadvertently capture global, systematic biases—such as language identity or domain context—that manifest as a constant vector offset. This offset shifts the entire distribution's center of gravity away from the origin, frustrating attempts to directly compare or align them.

Embedding centering solves this problem through a simple geometric correction: calculating the mean vector of a representation space and subtracting it from every individual vector. By shifting the centroids of disparate distributions to a common origin, this technique strips away contextual biases and exposes the shared semantic structure underneath.

The Geometry of Rotational Alignment

To align two distinct vector spaces, researchers frequently use linear transformations like Orthogonal Procrustes alignment, which rotates one space to match the orientation of another. However, a pure rotation operates around the origin $(0,0,\dots,0)$. If two vector spaces are offset from each other, rotating them around the origin will warp their internal semantic distances rather than aligning them cleanly.

Uncentered Spaces:                 Centered Spaces:
   (Language A)                       
       O  O                              O  O
      O  O                                 O  O
                                        /    \
               (Language B)            /  X   \  <-- Shared Origin
                   X  X               /        \
                  X  X                   X  X
                                        X  X

By applying dimension-wise mean centering as a preprocessing step, we shift the center of gravity of both spaces to the origin. As [Mikel Artetxe] et al. demonstrate in Learning principled bilingual mappings of word embeddings while preserving monolingual invariance, ensuring that the mean value of each dimension is zero maximizes the covariance between the spaces. This geometric reset makes subsequent rigid rotations highly accurate because the transformation focuses entirely on structural orientation rather than trying to bridge a spatial gap.

This principle holds true across diverse alignment tasks. When tracking how word meanings shift over time or across domains, [Dominik Schlechtweg] et al. show in A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains that centering vectors around their respective means before rotational alignment significantly improves mapping accuracy. Similarly, in the field of automated terminology extraction, [Anna Hätty] et al. establish in Predicting Degrees of Technicality in Automatic Terminology Extraction that column mean-centering makes general and domain-specific vector distributions more isotropic. This structural uniformity facilitates a cleaner mathematical comparison, allowing the direct vector difference between a word's general and technical representation to serve as a reliable proxy for its technicality.

Stripping Language Identity from Multilingual Models

The need for centering is perhaps most visible in massive multilingual language models like mBERT or XLM-RoBERTa. Although these models possess remarkable cross-lingual capabilities, they do not naturally map different languages to a single, language-agnostic space. Instead, they retain a strong "language identity" signal that acts as a constant spatial offset.

In On the Language Neutrality of Pre-trained Multilingual Representations, [Jindrich Libovický] et al. show that calculating the mean vector for a corpus in a given language and subtracting it from individual representations effectively neutralizes this language-specific bias. Removing this constant offset allows the underlying semantic spaces to achieve an unsupervised, near-perfect alignment.

This geometric correction is particularly vital when dealing with linguistically distant language pairs or low-resource scenarios:

Meeting in the Middle

Traditional alignment frameworks are typically unidirectional: they treat one language space as a fixed target and project the source language space directly onto it. This approach rests on a rigid and often flawed assumption that the target space possesses a perfect, unyielding geometry that the source language must strictly conform to.

To break this rigid assumption, [Yerai Doval] et al. introduce the "Meeting in the Middle" (Meemi) framework in Improving Cross-Lingual Word Embeddings by Meeting in the Middle. Instead of forcing a one-way projection, Meemi maps aligned vector pairs from both languages toward their shared arithmetic mean.

Unidirectional Mapping:          Meeting in the Middle (Meemi):
Language A ------> Language B     Language A ----> [ Joint Mean ] <---- Language B

As elaborated in their follow-up work, Meemi: A simple method for post-processing and integrating cross-lingual word embeddings, this post-processing step pulls translations toward a neutral, joint representation. Moving both spaces toward a common intermediate manifold relaxes strict structural constraints, which reduces language-specific biases while preserving the internal monolingual quality of both spaces.

This bidirectional adjustment strategy is also highly effective for contextualized embeddings. In Investigating Cross-Lingual Alignment Methods for Contextualized Embeddings with Token-Level Evaluation, [Qianchu Liu] demonstrates that shifting both source and target vectors toward a shared central representation after an initial projection significantly minimizes the distance between aligned token pairs.

Whether implemented as a zero-mean preprocessing step or a bidirectional post-processing adjustment, centering operates on a shared intuition: the truest semantic representation is found by stripping away the constant spatial offsets that isolate our data.

Key Foundational Work

Learning principled bilingual mappings of word embeddings while preserving monolingual invariance (2016) — contributed 2 concept elements to this direction.
Improving Cross-Lingual Word Embeddings by Meeting in the Middle (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_embedding_centering_06d2ba61,
  title = {embedding centering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-centering/rdr_06d2ba6134b14ecf8005ef6579d72442},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_embedding_centering_06d2ba61_v1,
  title = {embedding centering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/embedding-centering/rdr_06d2ba6134b14ecf8005ef6579d72442/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake