Introduction
Multilingual Language Models (ML-LMs), such as mBERT and XLM-R, have revolutionized cross-lingual NLP. However, these models often exhibit a significant "language identity" bias, where embeddings tend to cluster by language rather than semantic content. This phenomenon hinders zero-shot transfer, as the model may prioritize language-specific features over underlying meaning during tasks like cross-lingual retrieval or question answering. The challenge is to remove this language-specific signal without destroying the underlying meaning or requiring expensive parallel translation data.
Methodology: The LSAR Framework
To address this bias, we introduce LSAR (Low-rank Subspace for language-Agnostic Representations), an unsupervised, training-free post-processing technique. LSAR identifies and removes language-specific "noise" by projecting embeddings into a language-agnostic subspace.
The process consists of three main stages:
- Probing: Mean embeddings are extracted from multiple monolingual corpora (e.g., OSCAR) across a diverse set of languages.
- Identification: Singular Value Decomposition (SVD) is applied to these mean embeddings to identify the low-rank subspace where language-specific variance is most concentrated.
- Rectification: Original embeddings are projected into the null space of this identified language-specific subspace, effectively "subtracting" the language-specific components.
Unlike previous methods that handle languages individually, LSAR models the discrepancy across all languages jointly. This makes the rectification more robust and prevents the accidental removal of critical semantic information.
Key Insights: The Role of Syntax
A significant finding of this work is the nature of the information being removed. By performing a correlation analysis with the URIEL database, we discovered that the language-specific subspace is highly indicative of syntax (grammatical and structural relations). This suggests that ML-LMs primarily distinguish languages through their structural differences. By stripping away these syntactic signals, the model is forced to rely on "pure" semantic representations.
Experimental Results
LSAR was evaluated on several standard benchmarks, including Tatoeba (sentence retrieval) and LAReQA (cross-lingual QA retrieval). The method consistently outperformed both original models and existing de-biasing baselines.
- Sentence Retrieval: On mBERT, retrieval accuracy on Tatoeba increased from 37.53% to 44.64% (a ~19% relative improvement). On XLM-R, it improved from 57.68% to 65.05%.
- QA Retrieval: In the challenging XQuAD-R task, LSAR nearly doubled the mean Average Precision (mAP) for mBERT in the most difficult cross-lingual settings.
As shown in the visualization, LSAR effectively collapses language-specific clusters into a unified, language-agnostic space, facilitating better semantic alignment across different languages compared to the original embeddings or the LIR baseline.
Conclusion and Limitations
LSAR provides a powerful, "plug-and-play" solution for enhancing the language-agnosticism of ML-LMs without requiring parallel data or additional training. While highly effective for retrieval tasks, the gains are more modest in classification tasks like sentiment analysis, where some syntactic information may remain useful. Future work will explore moving beyond simple mean embeddings and integrating these low-rank discoveries directly into the pretraining phase of language models.
Summary updated