Abstract
Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. This study introduces a promising direction for enhancing non-English performance through a specialized pruning approach. Specifically, we prune MLLMs using bilingual sentence pairs from English and other languages and empirically demonstrate that this pruning strategy can enhance the MLLMs’ performance in non-English language.
Introduction
Despite being trained on diverse, multilingual datasets, state-of-the-art Multilingual Large Language Models (MLLMs) like XGLM and BLOOM often exhibit a pronounced "English-centric" bias. These models typically reason more effectively in English than in other languages such as Russian, Swahili, or Chinese. While "translate-test"—translating queries into English before processing—is a common workaround, it is computationally intensive and dependent on external translation quality. This research explores whether the inherent cross-lingual alignment within MLLMs can be unlocked through targeted pruning to improve non-English performance without external tools.
Methodology
The core observation driving this work is that specific "large magnitude features" (neurons with high activation values) are uniquely active during translation tasks but often dormant during standard monolingual tasks.
Feature Detection and Pruning
The researchers analyzed models like XGLM and mGPT to identify features active during few-shot translation demonstrations (e.g., "French: [text] English: [translation]"). They employed a modified version of Wanda (Pruning by Weights and Activations), using translation demonstrations as calibration data. Instead of traditional compression, pruning served as a "feature filter," removing weights that do not contribute to these high-magnitude alignment features.
Refinement for BLOOM
Initial experiments showed that BLOOM did not benefit from standard pruning, likely due to noise from its extensive training on programming code. The authors refined the pruning metric for BLOOM to specifically penalize features active during code generation while preserving those essential for translation.
Datasets
Calibration was performed using 400 bilingual sentence pairs from the FLORES-200 dataset. Evaluation was conducted on the XNLI (Cross-lingual Natural Language Inference) task across 15 languages and the MARC (Multilingual Amazon Review Corpus) for sentiment analysis.
Key Results
The study demonstrates that pruning can act as a "steering" mechanism. By removing 30% of the model's weights—specifically those that do not support cross-lingual alignment—the model is forced to utilize its most effective multilingual pathways.
As shown in the results table, performance improved across multiple architectures:
- XGLM-2.9B: Average XNLI accuracy increased from 42.5% to 44.5%, with significant gains in languages like Bulgarian (up to 45.8%).
- mGPT: Average XNLI accuracy rose from 39.6% to 40.3%.
- BLOOM-3B: Refined "code-aware" pruning boosted MARC sentiment analysis scores from 60.2 to 61.0.
Beyond raw accuracy, pruned models showed improved Cross-Lingual Consistency (measured by RankC), indicating they are more likely to provide the same answer regardless of the input language.
Discussion and Limitations
This research confirms the existence of "translation neurons"—high-magnitude features instrumental for cross-lingual tasks that appear consistently across different architectures. Furthermore, it highlights that for multilingual models, code-related features can act as interference for natural language reasoning.
However, several areas remain for future exploration. The optimal pruning ratio (currently fixed at 30%) may vary by language. The reliance on high-quality bilingual pairs for calibration could be a bottleneck for low-resource languages. Finally, as models scale to 70B+ parameters or undergo instruction tuning, the impact of this pruning strategy on already-aligned "frontier" models needs further investigation. There is also an ethical consideration: strengthening English-alignment might inadvertently facilitate the transfer of English-centric cultural biases into other languages.
Summary updated