Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates

Atsuki Yamaguchi, Terufumi Morishita, Aline Villavicencio, Nikolaos AletrasSubmission to ICLR 2026 · Withdrawn

TL;DR

Adapting instruction-tuned LLMs to low-resource languages using only raw, unlabeled text usually induces severe catastrophic forgetting, wrecking English instruction-following and safety alignment. Source-Shielded Updates (SSU) prevents this by calculating parameter importance on a small sample of source data and freezing the most critical structural columns before continual pre-training. Across 7B and 13B models, SSU protects core chat, reasoning, and safety capabilities while matching or outperforming full fine-tuning on the target language.

What Problem Does This Paper Address?

Expanding open-source instruct LLMs to underrepresented languages is difficult because high-quality instruction-tuning data in these languages is scarce or expensive to create. The most accessible route is continual pre-training (CPT) on raw, unlabeled target-language web text.

However, doing standard full fine-tuning (FFT) on raw text triggers severe catastrophic forgetting: the model loses its ability to follow instructions, solve math problems, stay safe, and converse cleanly in its original source language (typically English). Existing workarounds—such as LoRA, weight merging, or dynamic parameter selection based on target-data gradients—either fail to learn the new language well or still degrade core instruction-following abilities because unstructured target text pulls the weights away from conversational alignment.

Methods

SSU introduces a static, proactive, source-focused selective update strategy divided into three stages:

  1. Source-Driven Importance Scoring: SSU runs a tiny slice of English instruction data (e.g., 500 samples) through the model to score parameter importance. By default, it uses the Wanda pruning metric ($s_{ij} = |\theta_{ij}| \cdot \|X_j\|_2$), which highlights weights that have both large magnitudes and consistently high activation.
  2. Column-Wise Mask Generation: Rather than naive element-wise masking (which breaks coordinate transformations) or row-wise masking, SSU aggregates scores across columns ($S_j = \sum_i s_{ij}$). A weight column corresponds to a specific input feature pathway. SSU sets a binary mask that freezes the top $k\%$ most critical columns (50% by default).
  3. Continual Pre-Training (CPT): The model is trained on unlabeled target-language corpora via standard causal language modeling, zeroing out gradient updates for the shielded columns.
Input Features  ──► [ Column-wise Mask (Top-k% Frozen) ] ──► Preserved Feature Pathways
Target Gradients ──► [ Unfrozen Columns (Updated) ]        ──► New Language Acquisition

Datasets

  • Calibration: 500 samples from tulu-3-sft-olmo-2-mixture (and Stanford Alpaca for ablation).
  • Target CPT Data: Clean subset of MADLAD-400 (200M tokens per language).
  • Target Languages: Five typologically diverse, low-resource languages: Nepali (Devanagari), Kyrgyz (Cyrillic), Amharic (Ge’ez), Hausa (Latin), and Igbo (Latin).
  • Evaluation Benchmarks:
  • Instruction-Following & Reasoning (EN): IFEval, AlpacaEval 2.0 (AE2), MT-Bench, and GSM8K.
  • Safety Alignment (EN): Tulu 3 safety suite.
  • Multilingual / Cross-Lingual: FLORES-200 (Machine Translation), XL-SUM (Summarization), Belebele (Reading Comprehension), and MMLU / Global MMLU.

Key Insights and Ideas

  • Protecting Columns Preserves Pathways: Element-wise freezing breaks feature spaces, while row-wise freezing fails to preserve end-to-end output features. Column-wise freezing acts like protecting the load-bearing pillars of a building during renovation—retaining the foundational source transformations intact.
  • Source-Driven beats Target-Driven Selection: Dynamic methods that select parameters based on target gradients (like GMT) allow corrupting updates across the entire parameter space over time. In contrast, locking source-critical pathways beforehand enforces a hard subspace constraint that keeps instruction-following intact.
  • Fixing Linguistic Code-Mixing: When prompted in English after CPT, baseline models frequently respond in the target language or spit out jumbled code-mixed text (up to 16.9% on AE2 for GMT). SSU slashes code-mixed responses down to 1.0%, maintaining clean language boundaries.
  • Freezing Acts as Implicit Regularization: Restricting updates to 50% of the weights does not hurt target-language acquisition; instead, it stabilizes optimization, matching or exceeding dense full fine-tuning.

Key Results

Experiments conducted on OLMo 2 Instruct (7B and 13B) show:

  • Mitigating Forgetting: On monolingual source tasks, SSU drops only 3.4% (7B) and 2.8% (13B), compared to catastrophic drops of 20.3% and 22.3% under full fine-tuning (FFT).
  • Instruction-Following & Safety: SSU achieves near-source performance on IFEval, AE2, and MT-Bench (only a 5.9% and 4.7% relative drop for 7B/13B, versus >32% drop for FFT). Safety scores drop by a negligible 0.1% (7B) and 2.0% (13B).
  • Target Language Gains: SSU-Wanda outperforms FFT on all target benchmarks for the 7B model and on half for the 13B model (e.g., Igbo target MT jumps from 23.0 to 37.1 chrF++). In contrast, parameter-efficient baselines like AdaLoRA barely acquire the target language.
  • Robustness: SSU works equally well when using alternative importance scores (Fisher Information Matrix, SparseGPT) or alternative calibration sources (Stanford Alpaca).

Limitations and Future Work

  • Fixed Tokenizer Architecture: SSU assumes a fixed vocabulary and architecture. It does not address tokenization overfragmentation common in low-resource scripts; exploring the combination of SSU with vocabulary expansion remains future work.
  • Scale and Scope: Evaluations were limited to 7B and 13B models from the OLMo 2 family. Validating the technique on larger scales (>70B) and Mixture-of-Experts (MoE) architectures is an open direction.
  • Hyperparameter Trade-off: There is an inherent Pareto frontier governed by the freezing ratio $k$: higher ratios preserve source capabilities better, whereas lower ratios facilitate greater target language adaptation.

Summary updated

Abstract

Expanding the linguistic diversity of instruct large language models (LLMs) is crucial for global accessibility but is often hindered by the reliance on costly specialized target language labeled data and catastrophic forgetting during adaptation. We tackle this challenge under a realistic, low-resource constraint: adapting instruct LLMs using only unlabeled target language data. We introduce **S**ource-**S**hielded **U**pdates (**SSU**), a selective parameter update strategy that proactively preserves source knowledge. Using a small set of source data and a parameter importance scoring method, SSU identifies parameters critical to maintaining source abilities. It then applies a column-wise freezing strategy to protect these parameters before adaptation. Experiments across five typologically diverse languages and 7B and 13B models demonstrate that SSU successfully mitigates catastrophic forgetting. It reduces performance degradation on monolingual source tasks to just 3.4% (7B) and 2.8% (13B) on average, a stark contrast to the 20.2% and 22.3% from full fine-tuning. SSU also achieves target-language performance highly competitive with full fine-tuning, outperforming it on all benchmarks for 7B models and the majority for 13B models.
BibTeX
@misc{yamaguchi2025mitigating,
  author = {Atsuki Yamaguchi and Terufumi Morishita and Aline Villavicencio and Nikolaos Aletras},
  title = {{Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates}},
  year = {2025},
  eprint = {2512.04844},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04844},
}
Figure