The Geometry of Multilingual Language Model Representations

Tyler A. Chang, Zhuowen Tu, Benjamin K. Bergen

TL;DR

Multilingual language models like XLM-R don't just jumble all languages into a single shared space; instead, they align them into nearly parallel linear subspaces separated by language-specific means. By decomposing these spaces, the authors reveal that models simultaneously encode language identity and structural features (like part-of-speech and token position) along orthogonal axes, forming striking geometric structures like spirals and toruses.

What Problem Does This Paper Address?

State-of-the-art multilingual language models can process dozens of languages simultaneously and transfer knowledge across them seamlessly. The common intuition is that these models map everything into a "shared multilingual embedding space."

However, this creates a paradox: if the space is completely shared and language-neutral, how does the model retain language-specific nuances, typological traits, and vocabulary? The authors tackle this fundamental question by exploring the underlying geometry of multilingual representation spaces, investigating how language-sensitive and language-neutral information coexist without interfering with each other.

Methods

Using the pre-trained XLM-R (base) model as a case study, the authors employ a mix of geometric analysis and causal interventions:

  1. Affine Subspace Identification: They use Singular Value Decomposition (SVD) on contextualized token representations to define a mean-centered linear subspace for 88 different languages.
  2. Causal Projections: To prove these subspaces actually matter, they project representations onto specific language subspaces and measure the impact on downstream language modeling performance (perplexity).
  3. Subspace Distance Metric: They adopt a Riemannian metric designed for positive definite matrices to directly calculate the distance between language covariance matrices, translating abstract distances into intuitive "rotation degrees" and "scaling multipliers."
  4. Linear Discriminant Analysis (LDA): They use LDA to explicitly find the axes that maximize the separation between languages (language-sensitive axes) or between structural properties like token positions and parts-of-speech (language-neutral axes).

Datasets

The study utilizes two primary datasets:

  • OSCAR corpus: A cleaned web-text dataset used to extract 262K contextualized token representations for 88 languages that overlap with XLM-R's pre-training data.
  • Universal Dependencies (UD) dataset: Used to map tokens to their respective part-of-speech (POS) tags across 61 of the available languages.

Key Insights and Ideas

  • Parallel Subspaces: After subtracting the language-specific mean vector, the underlying linear subspaces for different languages are remarkably similar. They are essentially parallel tracks shifted by a language bias vector.
  • The Orthogonality Principle: Language-sensitive features (like vocabulary and language family) and language-neutral features (like syntax and position) are encoded along orthogonal axes. This means you can look at a representation from one angle to see what is being said, and from another angle to see which language it's being said in.
  • Geometric Manifolds for Position: Token position isn't just a flat index. The model encodes absolute position along curves/spirals, and relative position (modulo a window size) within a nearly perfect torus.
Figure 1: XLM-R representations in layer six projected onto a linear subspace where two axes are language- neutral (horizontal axes), and one axis is language- sensitive (vertical axis). Projecting from the side visual- izes the language-sensitive axis (top right). Projecting from the top down visualizes the language-neutral axes encoding token position information, forming a nearly perfect torus in the representation space (bottom right).
  • Layer Stability: These subspaces and axes are surprisingly stable throughout the middle layers (layers 5 to 9), implying that the model maintains a consistent internal coordinate system while performing higher-level semantic processing.

Key Results

  • Perplexity and Mean-Shifting: Projecting a language's representations onto its own 335-dimensional SVD subspace (compressing from 768 dimensions) causes almost no drop in language modeling performance. Projecting Language A onto Language B's subspace destroys performance—unless you shift Language B's subspace to pass through Language A's mean, which completely restores performance in the middle layers.
Language modeling perplexity scores across different layer projections
  • Subspace Closeness: In middle layers, the mean rotation between any two language subspaces is less than 5 degrees, demonstrating extreme cross-lingual alignment once means are centered.
Analogous rotations and scalings between mean-centered language subspaces
  • Inducing Target Vocabularies: By simply adding a shift vector ($\mu_B - \mu_A$) and projecting onto subspace B, the authors successfully manipulated the model into predicting tokens in target language B (increasing target token generation from 10% to 47%), proving the causal relevance of these geometric axes.

Limitations and Future Work

  • Linguistic Bias: The 88 languages analyzed are heavily skewed toward Indo-European languages (52 out of 88). The dataset completely lacks Native American languages and features only two African languages, meaning the cross-lingual similarity might be overestimated.
  • Confounding Corpus Topics: The language-sensitive axes might inadvertently capture shifts in topic distributions across different language corpora rather than purely linguistic features.
  • Single Model Evaluation: The experiments were exclusively performed on XLM-R. Future work is required to verify if these geometric properties hold true across different architectures (like causal LLMs) and training objectives.

Summary updated

Abstract

We assess how multilingual language models maintain a shared multilingual representation space while still encoding language-sensitive information in each language. Using XLM-R as a case study, we show that languages occupy similar linear subspaces after mean-centering, evaluated based on causal effects on language modeling performance and direct comparisons between subspaces for 88 languages. The subspace means differ along language-sensitive axes that are relatively stable throughout middle layers, and these axes encode information such as token vocabularies. Shifting representations by language means is sufficient to induce token predictions in different languages. However, we also identify stable language-neutral axes that encode information such as token positions and part-of-speech. We visualize representations projected onto language-sensitive and language-neutral axes, identifying language family and part-of-speech clusters, along with spirals, toruses, and curves representing token position information. These results demonstrate that multilingual language models encode information along orthogonal language-sensitive and language-neutral axes, allowing the models to extract a variety of features for downstream tasks and cross-lingual transfer learning.
BibTeX
@inproceedings{chang2022geometry,
  author = {Tyler A. Chang and Zhuowen Tu and Benjamin K. Bergen},
  title = {{The Geometry of Multilingual Language Model Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.9},
}
Figure