Abstract
Adapting instruction-tuned Large Language Models (LLMs) to new, low-resource languages often results in "catastrophic forgetting," where the model loses its ability to follow instructions or reason logically. We present Source-Shielded Updates (SSU), a proactive method that identifies and freezes critical parameter columns using source-language calibration data before adaptation begins. SSU effectively preserves core model functionality while achieving target-language proficiency that often exceeds standard full fine-tuning.
Introduction
When an "instruct" model is adapted to a new language (e.g., Igbo or Kyrgyz) using unlabeled text, the signals from the raw, unstructured target data are often misaligned with the structured logic learned during initial instruction tuning. This misalignment causes the model to lose its "intelligence"—its capacity for complex reasoning and adherence to safety guardrails. While existing solutions often attempt to fix this post-hoc through model merging, SSU addresses the issue during the adaptation process itself.
The SSU Method
Source-Shielded Updates (SSU) follows a three-stage proactive strategy to protect the model's existing knowledge base while allowing for new linguistic acquisition.
- Importance Scoring: Before training, the model processes a small sample (500 instances) of its original English instruction data. Using the Wanda metric—which combines weight magnitude with the norm of input activations—SSU identifies the parameters most critical for maintaining core abilities.
- Column-wise Masking: Rather than freezing individual weights, SSU aggregates scores to freeze entire columns of weight matrices. This preserves the structural "feature pathways" within the model, keeping its internal logic stable.
- Continual Pre-training (CPT): The model is trained on the target language data with a static mask applied to gradients. Typically, the top 50% most important columns are shielded from updates, while the remaining parameters adapt to the new language.
Experimental Setup
The authors evaluated SSU using OLMo 2 (7B and 13B) models. Calibration was performed using English instruction data (tulu-3-sft-olmo-2-mixture). The target data consisted of unlabeled text for five diverse, low-resource languages: Nepali, Kyrgyz, Amharic, Hausa, and Igbo. Performance was measured across a wide array of benchmarks, including IFEval for instruction following, GSM8K for math, and FLORES-200 for translation.
Results
Empirical results demonstrate that SSU significantly outperforms traditional adaptation methods in both knowledge preservation and new language acquisition.
- Preserving the "Brain": SSU reduced performance degradation on English tasks to just 3.4% (7B) and 2.8% (13B). In contrast, standard full fine-tuning (FFT) saw massive drops exceeding 20%.
- Target Proficiency: SSU outperformed FFT on all target-language benchmarks for 7B models. By freezing 50% of the model, SSU acts as a regularizer, preventing overfitting to noisy, raw text.
- Safety Alignment: SSU almost perfectly preserved safety guardrails (0.1% - 2.0% drop), whereas other methods saw safety scores plummet by over 10%.
Discussion and Insights
A key finding is that column-wise freezing is superior to element-wise freezing. Freezing random individual weights can corrupt mathematical transformations, whereas freezing columns preserves the entire processing pathway for a feature. Furthermore, SSU's "source-focused" approach—looking at old data to decide what to save rather than new data to decide what to change—is more effective at preventing the model from becoming a "zombie" that knows a language but cannot follow a prompt.
Limitations
While effective, SSU currently focuses on parameter updates within a fixed architecture and does not address vocabulary adaptation. Additionally, while it only requires 500 samples, it still necessitates access to a portion of the original source/instruction data, which may be a constraint for proprietary models. Future work involves optimizing the freezing ratio for specific use cases, as higher ratios better preserve source knowledge while lower ratios can slightly enhance target acquisition.
Summary updated