Abstract
Large pretrained multilingual language models (ML-LMs) have shown remarkable capabilities of zero-shot cross-lingual transfer, without direct cross-lingual supervision. While these results are promising, follow-up works found that, within the multilingual embedding spaces, there exists strong language identity information which hinders the expression of linguistic factors shared across languages. For semantic tasks like cross-lingual sentence retrieval, it is desired to remove such language identity signals to fully leverage semantic information. In this work, we provide a novel view of projecting away language-specific factors from a multilingual embedding space. Specifically, we discover that there exists a low-rank subspace that primarily encodes information irrelevant to semantics (e.g., syntactic information). To identify this subspace, we present a simple but effective unsupervised method based on singular value decomposition with multiple monolingual corpora as input. Once the subspace is found, we can directly project the original embeddings into the null space to boost language agnosticism without finetuning. We systematically evaluate our method on various tasks including the challenging language-agnostic QA retrieval task. Empirical results show that applying our method consistently leads to improvements over commonly used ML-LMs.
Introduction
Pretrained multilingual language models (ML-LMs) such as mBERT and XLM-R have revolutionized zero-shot cross-lingual transfer. Despite their success, these models suffer from a persistent "language identity" bias. In the high-dimensional embedding space, representations tend to cluster by the language of the input text rather than the underlying semantic concepts. For instance, the concept of "gravity" in English may be positioned far from its Spanish equivalent simply because the model prioritizes language-specific features over semantic ones. This clustering hinders performance in critical cross-lingual tasks like sentence retrieval and question answering.
Methodology: The LSAR Framework
We propose LSAR (Low-rank Subspace for language-Agnostic Representations), an unsupervised post-hoc method designed to rectify this bias. The core hypothesis is that language-specific noise—such as syntax and script—is confined to a low-rank subspace that can be mathematically isolated and removed.
The LSAR process involves two main steps:
- Subspace Identification: We collect mean embeddings from multiple monolingual corpora (e.g., OSCAR, Wiki-40B). Unlike previous methods, LSAR does not require parallel translation pairs. We apply Singular Value Decomposition (SVD) to these mean vectors to identify a low-rank matrix ($M_s$) that captures the primary directions of cross-lingual variance.
- Rectification via Projection: Once the language-specific subspace is identified, we project the original embeddings into its null space. This effectively "erases" the dimensions responsible for language identity, forcing the model to rely on shared semantic features.
Experimental Evaluation
We evaluated LSAR across a diverse set of benchmarks to test its impact on semantic alignment, including Tatoeba for sentence retrieval, LAReQA (XQuAD-R and MLQA-R) for language-agnostic question-answer retrieval, and Amazon Reviews for zero-shot sentiment classification.
Key Results
The application of LSAR yielded significant performance gains across all tested models:
- Retrieval Accuracy: On the Tatoeba benchmark, LSAR improved mBERT’s retrieval accuracy by up to 19% (relative).
- Question Answering: In the challenging LAReQA task, LSAR nearly doubled the mean Average Precision (mAP) for mBERT on the XQuAD-R subset.
- Robustness: LSAR consistently outperformed simpler methods like mean centering or per-language principal component removal (LIR), proving that the joint low-rank approach is more stable.
Discussion and Insights
The Role of Syntax
Correlation analysis using the URIEL database revealed that the subspace removed by LSAR is heavily encoded with syntactic information. This suggests that while syntax is a defining characteristic of individual languages, it acts as "noise" in semantic retrieval tasks. By removing this syntactic layer, LSAR allows the model to focus on universal semantic signals shared across the linguistic spectrum.
Global Alignment vs. Local Normalization
Traditional methods often attempt to normalize each language's embedding space individually. In contrast, LSAR adopts a "global" view by analyzing the relationships between many languages simultaneously. This joint low-rank approach proves more effective at isolating universal semantic signals from language-specific noise.
Comparison with Supervised Alignment
Interestingly, LSAR provides the most significant boosts to models like mBERT and XLM-R, while showing smaller gains for models like LaBSE. Since LaBSE is already trained on massive parallel datasets to be language-agnostic, the fact that LSAR brings mBERT closer to LaBSE's performance suggests that our unsupervised method successfully uncovers the same semantic alignments that supervised training provides.
Conclusion and Future Work
LSAR demonstrates that language-specific information in ML-LMs is low-rank and can be removed post-hoc to significantly improve cross-lingual transfer. While highly effective, the current implementation relies on first-moment (mean) statistics. Future work could explore incorporating higher-order statistics, such as covariance, to capture more complex language-specific factors. Additionally, integrating the low-rank subspace constraint directly into the pretraining objective could lead to inherently more language-agnostic models.
Summary updated