Abstract
Multilingual Large Language Models (MLLMs) demonstrate impressive zero-shot capabilities across diverse languages but continue to exhibit a significant performance gap between English and non-English tasks. This study investigates the internal mechanisms of MLLMs during translation and identifies a subset of "large magnitude features" that are critical for cross-lingual alignment. By employing a surgical pruning strategy that preserves these features while removing redundant weights, we demonstrate that MLLMs can be "steered" to improve their zero-shot performance in non-English languages without the need for external translation tools or expensive fine-tuning.
Introduction
While models like XGLM and BLOOM are trained on balanced multilingual data, they consistently perform better in English. Traditional methods to bridge this gap, such as "translate-test" pipelines, are computationally intensive and rely on external systems. This research explores whether a model's inherent alignment—the internal logic it uses to connect English concepts to other languages—can be unlocked through strategic pruning.
Methodology: Pruning as a Steering Mechanism
The researchers utilized Wanda (Pruning by Weights and Activations) not just to reduce model size, but as a tool to refine model behavior. The process involves three key steps:
- Feature Detection: Analysis of hidden states revealed that few-shot translation prompts (e.g., "English: Hello, French: Bonjour") activate specific "large magnitude features" that remain dormant during standard monolingual tasks.
- Calibration-based Pruning: Instead of generic data, the authors used bilingual sentence pairs from FLORES-200 as calibration data. They retained weights associated with these high-magnitude translation features and pruned 30% of the remaining weights.
- Refinement for Code-Heavy Models: For models like BLOOM, which include programming data, the authors introduced a metric to penalize features active during code generation, effectively silencing "coding noise" to enhance natural language alignment.
Key Insights
- Translation as a Distinct Mode: Translation activates a specific set of neurons that are largely independent of the language pair, suggesting a universal internal alignment mechanism.
- Enhanced Consistency: Using the RankC metric, the study found that pruned models provide more consistent answers across different languages, indicating a deeper logical alignment rather than surface-level guessing.
- The "Noise" of Code: In models like BLOOM, code-generation features can interfere with linguistic reasoning. Pruning these "distractor" weights unlocked better multilingual performance.
Experimental Results
The pruning strategy was evaluated on XNLI (Cross-lingual Natural Language Inference) and MARC (Multilingual Amazon Review Corpus) across models including XGLM (2.9B), mGPT (1.7B), and BLOOM (3B).
- XGLM-2.9B: Translation-feature pruning improved average XNLI accuracy from 42.5 to 44.5, with notable gains in Bulgarian and Greek.
- mGPT-1.7B: Average XNLI accuracy rose from 39.6 to 40.3.
- BLOOM-3B: Refined "anti-code" pruning improved MARC scores from 60.2 to 61.0.
Conclusion and Future Work
This research demonstrates that a leaner model can be more effective if the right components are preserved. By forcing the model to rely on its high-quality English reasoning pathways for non-English inputs, pruning acts as a steering mechanism for better multilingual inference. Future work will explore the optimal pruning ratios and address the potential for transferring English-centric biases through strengthened alignment.
Summary updated