Pruning Multilingual Large Language Models for Multilingual Inference

Hwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru Komachi

Abstract

Multilingual Large Language Models (MLLMs) demonstrate impressive zero-shot capabilities across diverse languages but continue to exhibit a significant performance gap between English and non-English tasks. This study investigates the internal mechanisms of MLLMs during translation and identifies a subset of "large magnitude features" that are critical for cross-lingual alignment. By employing a surgical pruning strategy that preserves these features while removing redundant weights, we demonstrate that MLLMs can be "steered" to improve their zero-shot performance in non-English languages without the need for external translation tools or expensive fine-tuning.

Introduction

While models like XGLM and BLOOM are trained on balanced multilingual data, they consistently perform better in English. Traditional methods to bridge this gap, such as "translate-test" pipelines, are computationally intensive and rely on external systems. This research explores whether a model's inherent alignment—the internal logic it uses to connect English concepts to other languages—can be unlocked through strategic pruning.

Methodology: Pruning as a Steering Mechanism

The researchers utilized Wanda (Pruning by Weights and Activations) not just to reduce model size, but as a tool to refine model behavior. The process involves three key steps:

  1. Feature Detection: Analysis of hidden states revealed that few-shot translation prompts (e.g., "English: Hello, French: Bonjour") activate specific "large magnitude features" that remain dormant during standard monolingual tasks.
  2. Calibration-based Pruning: Instead of generic data, the authors used bilingual sentence pairs from FLORES-200 as calibration data. They retained weights associated with these high-magnitude translation features and pruned 30% of the remaining weights.
  3. Refinement for Code-Heavy Models: For models like BLOOM, which include programming data, the authors introduced a metric to penalize features active during code generation, effectively silencing "coding noise" to enhance natural language alignment.
The magnitude distribution of features in MLLMs shows that a small number of features possess significantly higher magnitudes, which are targeted for preservation during pruning.

Key Insights

  • Translation as a Distinct Mode: Translation activates a specific set of neurons that are largely independent of the language pair, suggesting a universal internal alignment mechanism.
  • Enhanced Consistency: Using the RankC metric, the study found that pruned models provide more consistent answers across different languages, indicating a deeper logical alignment rather than surface-level guessing.
  • The "Noise" of Code: In models like BLOOM, code-generation features can interfere with linguistic reasoning. Pruning these "distractor" weights unlocked better multilingual performance.

Experimental Results

The pruning strategy was evaluated on XNLI (Cross-lingual Natural Language Inference) and MARC (Multilingual Amazon Review Corpus) across models including XGLM (2.9B), mGPT (1.7B), and BLOOM (3B).

  • XGLM-2.9B: Translation-feature pruning improved average XNLI accuracy from 42.5 to 44.5, with notable gains in Bulgarian and Greek.
  • mGPT-1.7B: Average XNLI accuracy rose from 39.6 to 40.3.
  • BLOOM-3B: Refined "anti-code" pruning improved MARC scores from 60.2 to 61.0.
Performance comparison across different pruning strategies and source languages, demonstrating that translation-based calibration (bottom rows) often outperforms standard monolingual pruning.

Conclusion and Future Work

This research demonstrates that a leaner model can be more effective if the right components are preserved. By forcing the model to rely on its high-quality English reasoning pathways for non-English inputs, pruning acts as a steering mechanism for better multilingual inference. Future work will explore the optimal pruning ratios and address the potential for transferring English-centric biases through strengthened alignment.

Summary updated

Abstract

Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.
BibTeX
@inproceedings{kim2024pruning,
  author = {Hwichan Kim and Jun Suzuki and Tosho Hirasawa and Mamoru Komachi},
  title = {{Pruning Multilingual Large Language Models for Multilingual Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=9wl7jcszqO},
}
Figure