Pruning Multilingual Large Language Models for Multilingual Inference

Hwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru Komachi

Abstract

Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. This study introduces a promising direction for enhancing non-English performance through a specialized pruning approach. Specifically, we prune MLLMs using bilingual sentence pairs from English and other languages and empirically demonstrate that this pruning strategy can enhance the MLLMs’ performance in non-English language.

Introduction

Despite being trained on diverse, multilingual datasets, state-of-the-art Multilingual Large Language Models (MLLMs) like XGLM and BLOOM often exhibit a pronounced "English-centric" bias. These models typically reason more effectively in English than in other languages such as Russian, Swahili, or Chinese. While "translate-test"—translating queries into English before processing—is a common workaround, it is computationally intensive and dependent on external translation quality. This research explores whether the inherent cross-lingual alignment within MLLMs can be unlocked through targeted pruning to improve non-English performance without external tools.

Methodology

The core observation driving this work is that specific "large magnitude features" (neurons with high activation values) are uniquely active during translation tasks but often dormant during standard monolingual tasks.

Magnitude distribution of features, showing a small number of high-activation neurons.

Feature Detection and Pruning

The researchers analyzed models like XGLM and mGPT to identify features active during few-shot translation demonstrations (e.g., "French: [text] English: [translation]"). They employed a modified version of Wanda (Pruning by Weights and Activations), using translation demonstrations as calibration data. Instead of traditional compression, pruning served as a "feature filter," removing weights that do not contribute to these high-magnitude alignment features.

Refinement for BLOOM

Initial experiments showed that BLOOM did not benefit from standard pruning, likely due to noise from its extensive training on programming code. The authors refined the pruning metric for BLOOM to specifically penalize features active during code generation while preserving those essential for translation.

Datasets

Calibration was performed using 400 bilingual sentence pairs from the FLORES-200 dataset. Evaluation was conducted on the XNLI (Cross-lingual Natural Language Inference) task across 15 languages and the MARC (Multilingual Amazon Review Corpus) for sentiment analysis.

Key Results

The study demonstrates that pruning can act as a "steering" mechanism. By removing 30% of the model's weights—specifically those that do not support cross-lingual alignment—the model is forced to utilize its most effective multilingual pathways.

XNLI performance across models and languages.

As shown in the results table, performance improved across multiple architectures:

  • XGLM-2.9B: Average XNLI accuracy increased from 42.5% to 44.5%, with significant gains in languages like Bulgarian (up to 45.8%).
  • mGPT: Average XNLI accuracy rose from 39.6% to 40.3%.
  • BLOOM-3B: Refined "code-aware" pruning boosted MARC sentiment analysis scores from 60.2 to 61.0.

Beyond raw accuracy, pruned models showed improved Cross-Lingual Consistency (measured by RankC), indicating they are more likely to provide the same answer regardless of the input language.

Discussion and Limitations

This research confirms the existence of "translation neurons"—high-magnitude features instrumental for cross-lingual tasks that appear consistently across different architectures. Furthermore, it highlights that for multilingual models, code-related features can act as interference for natural language reasoning.

However, several areas remain for future exploration. The optimal pruning ratio (currently fixed at 30%) may vary by language. The reliance on high-quality bilingual pairs for calibration could be a bottleneck for low-resource languages. Finally, as models scale to 70B+ parameters or undergo instruction tuning, the impact of this pruning strategy on already-aligned "frontier" models needs further investigation. There is also an ethical consideration: strengthening English-alignment might inadvertently facilitate the transfer of English-centric cultural biases into other languages.

Summary updated

Abstract

Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. This study introduces a promising direction for enhancing non-English performance through a specialized pruning approach. Specifically, we prune MLLMs using bilingual sentence pairs from English and other languages and empirically demonstrate that this pruning strategy can enhance the MLLMs’ performance in non-English language.
BibTeX
@inproceedings{kim2024pruning,
  author = {Hwichan Kim and Jun Suzuki and Tosho Hirasawa and Mamoru Komachi},
  title = {{Pruning Multilingual Large Language Models for Multilingual Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=9wl7jcszqO},
}
Figure