TL;DR
Multilingual language models like XLM-R don't just jumble all languages into a single shared space; instead, they align them into nearly parallel linear subspaces separated by language-specific means. By decomposing these spaces, the authors reveal that models simultaneously encode language identity and structural features (like part-of-speech and token position) along orthogonal axes, forming striking geometric structures like spirals and toruses.
What Problem Does This Paper Address?
State-of-the-art multilingual language models can process dozens of languages simultaneously and transfer knowledge across them seamlessly. The common intuition is that these models map everything into a "shared multilingual embedding space."
However, this creates a paradox: if the space is completely shared and language-neutral, how does the model retain language-specific nuances, typological traits, and vocabulary? The authors tackle this fundamental question by exploring the underlying geometry of multilingual representation spaces, investigating how language-sensitive and language-neutral information coexist without interfering with each other.
Methods
Using the pre-trained XLM-R (base) model as a case study, the authors employ a mix of geometric analysis and causal interventions:
- Affine Subspace Identification: They use Singular Value Decomposition (SVD) on contextualized token representations to define a mean-centered linear subspace for 88 different languages.
- Causal Projections: To prove these subspaces actually matter, they project representations onto specific language subspaces and measure the impact on downstream language modeling performance (perplexity).
- Subspace Distance Metric: They adopt a Riemannian metric designed for positive definite matrices to directly calculate the distance between language covariance matrices, translating abstract distances into intuitive "rotation degrees" and "scaling multipliers."
- Linear Discriminant Analysis (LDA): They use LDA to explicitly find the axes that maximize the separation between languages (language-sensitive axes) or between structural properties like token positions and parts-of-speech (language-neutral axes).
Datasets
The study utilizes two primary datasets:
- OSCAR corpus: A cleaned web-text dataset used to extract 262K contextualized token representations for 88 languages that overlap with XLM-R's pre-training data.
- Universal Dependencies (UD) dataset: Used to map tokens to their respective part-of-speech (POS) tags across 61 of the available languages.
Key Insights and Ideas
- Parallel Subspaces: After subtracting the language-specific mean vector, the underlying linear subspaces for different languages are remarkably similar. They are essentially parallel tracks shifted by a language bias vector.
- The Orthogonality Principle: Language-sensitive features (like vocabulary and language family) and language-neutral features (like syntax and position) are encoded along orthogonal axes. This means you can look at a representation from one angle to see what is being said, and from another angle to see which language it's being said in.
- Geometric Manifolds for Position: Token position isn't just a flat index. The model encodes absolute position along curves/spirals, and relative position (modulo a window size) within a nearly perfect torus.
- Layer Stability: These subspaces and axes are surprisingly stable throughout the middle layers (layers 5 to 9), implying that the model maintains a consistent internal coordinate system while performing higher-level semantic processing.
Key Results
- Perplexity and Mean-Shifting: Projecting a language's representations onto its own 335-dimensional SVD subspace (compressing from 768 dimensions) causes almost no drop in language modeling performance. Projecting Language A onto Language B's subspace destroys performance—unless you shift Language B's subspace to pass through Language A's mean, which completely restores performance in the middle layers.
- Subspace Closeness: In middle layers, the mean rotation between any two language subspaces is less than 5 degrees, demonstrating extreme cross-lingual alignment once means are centered.
- Inducing Target Vocabularies: By simply adding a shift vector ($\mu_B - \mu_A$) and projecting onto subspace B, the authors successfully manipulated the model into predicting tokens in target language B (increasing target token generation from 10% to 47%), proving the causal relevance of these geometric axes.
Limitations and Future Work
- Linguistic Bias: The 88 languages analyzed are heavily skewed toward Indo-European languages (52 out of 88). The dataset completely lacks Native American languages and features only two African languages, meaning the cross-lingual similarity might be overestimated.
- Confounding Corpus Topics: The language-sensitive axes might inadvertently capture shifts in topic distributions across different language corpora rather than purely linguistic features.
- Single Model Evaluation: The experiments were exclusively performed on XLM-R. Future work is required to verify if these geometric properties hold true across different architectures (like causal LLMs) and training objectives.
Summary updated