Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations

Zhihui Xie, Handong Zhao, Tong Yu, Shuai Li

Introduction

Multilingual Language Models (ML-LMs), such as mBERT and XLM-R, have revolutionized cross-lingual NLP. However, these models often exhibit a significant "language identity" bias, where embeddings tend to cluster by language rather than semantic content. This phenomenon hinders zero-shot transfer, as the model may prioritize language-specific features over underlying meaning during tasks like cross-lingual retrieval or question answering. The challenge is to remove this language-specific signal without destroying the underlying meaning or requiring expensive parallel translation data.

Methodology: The LSAR Framework

To address this bias, we introduce LSAR (Low-rank Subspace for language-Agnostic Representations), an unsupervised, training-free post-processing technique. LSAR identifies and removes language-specific "noise" by projecting embeddings into a language-agnostic subspace.

The process consists of three main stages:

  1. Probing: Mean embeddings are extracted from multiple monolingual corpora (e.g., OSCAR) across a diverse set of languages.
  2. Identification: Singular Value Decomposition (SVD) is applied to these mean embeddings to identify the low-rank subspace where language-specific variance is most concentrated.
  3. Rectification: Original embeddings are projected into the null space of this identified language-specific subspace, effectively "subtracting" the language-specific components.
Overview of the LSAR framework, illustrating the probing, identification, and rectification stages.

Unlike previous methods that handle languages individually, LSAR models the discrepancy across all languages jointly. This makes the rectification more robust and prevents the accidental removal of critical semantic information.

Key Insights: The Role of Syntax

A significant finding of this work is the nature of the information being removed. By performing a correlation analysis with the URIEL database, we discovered that the language-specific subspace is highly indicative of syntax (grammatical and structural relations). This suggests that ML-LMs primarily distinguish languages through their structural differences. By stripping away these syntactic signals, the model is forced to rely on "pure" semantic representations.

Experimental Results

LSAR was evaluated on several standard benchmarks, including Tatoeba (sentence retrieval) and LAReQA (cross-lingual QA retrieval). The method consistently outperformed both original models and existing de-biasing baselines.

  • Sentence Retrieval: On mBERT, retrieval accuracy on Tatoeba increased from 37.53% to 44.64% (a ~19% relative improvement). On XLM-R, it improved from 57.68% to 65.05%.
  • QA Retrieval: In the challenging XQuAD-R task, LSAR nearly doubled the mean Average Precision (mAP) for mBERT in the most difficult cross-lingual settings.
Visualization of embeddings before and after rectification, comparing the original space with LIR and the proposed LSAR method.

As shown in the visualization, LSAR effectively collapses language-specific clusters into a unified, language-agnostic space, facilitating better semantic alignment across different languages compared to the original embeddings or the LIR baseline.

Conclusion and Limitations

LSAR provides a powerful, "plug-and-play" solution for enhancing the language-agnosticism of ML-LMs without requiring parallel data or additional training. While highly effective for retrieval tasks, the gains are more modest in classification tasks like sentiment analysis, where some syntactic information may remain useful. Future work will explore moving beyond simple mean embeddings and integrating these low-rank discoveries directly into the pretraining phase of language models.

Summary updated

Abstract

Large pretrained multilingual language models (ML-LMs) have shown remarkable capabilities of zero-shot cross-lingual transfer, without direct cross-lingual supervision. While these results are promising, follow-up works found that, within the multilingual embedding spaces, there exists strong language identity information which hinders the expression of linguistic factors shared across languages. For semantic tasks like cross-lingual sentence retrieval, it is desired to remove such language identity signals to fully leverage semantic information. In this work, we provide a novel view of projecting away language-specific factors from a multilingual embedding space. Specifically, we discover that there exists a low-rank subspace that primarily encodes information irrelevant to semantics (e.g., syntactic information). To identify this subspace, we present a simple but effective unsupervised method based on singular value decomposition with multiple monolingual corpora as input. Once the subspace is found, we can directly project the original embeddings into the null space to boost language agnosticism without finetuning. We systematically evaluate our method on various tasks including the challenging language-agnostic QA retrieval task. Empirical results show that applying our method consistently leads to improvements over commonly used ML-LMs.
BibTeX
@inproceedings{xie2022discovering,
  author = {Zhihui Xie and Handong Zhao and Tong Yu and Shuai Li},
  title = {{Discovering Low-rank Subspaces for Language-agnostic Multilingual Representations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=QueXTeFODd},
}
Figure