TL;DR
Adapting Large Language Models (LLMs) when data is scarce requires navigating a sharp trade-off between skill acquisition and knowledge retention. Comparing Supervised Fine-Tuning (SFT), Low-Rank Adaptation (LoRA), and In-Context Learning (ICL) on Gemma 3 reveals that while SFT learns rapidly from minimal examples, it triggers catastrophic forgetting; ICL completely preserves knowledge but fails at complex reasoning; and LoRA offers the most balanced compromise, provided enough data is available to form an effective low-rank update.
What Problem Does This Paper Address?
When adapting pre-trained LLMs to specialized domains or novel workflows, practitioners typically turn to one of three paradigms: full fine-tuning (SFT), parameter-efficient fine-tuning (PEFT via LoRA), or prompt-based in-context learning (ICL). In data-scarce settings—where only tens or hundreds of labeled examples are available—it remains unclear which method best acquires new capabilities without overwriting the model's pre-trained knowledge and general instruction-following abilities (known as catastrophic forgetting).
Methods
The authors conduct a controlled, head-to-head empirical comparison of standard adaptation methods using Google’s Gemma 3 model within the Kauldron framework:
- In-Context Learning (ICL): Few-to-many shot prompting (up to 128 shots, fitting within context lengths up to 32k tokens) without gradient updates.
- Supervised Fine-Tuning (SFT): Full parameter updates across learning rates ($10^{-3}$ to $10^{-4}$) and sample sizes (16 to 1024).
- Low-Rank Adaptation (LoRA): Low-rank matrix updates evaluated across varying ranks ($r \in \{1, 4, 16\}$) and sample sizes (8 to 8192), using a standard learning rate of $0.005$.
- Weight Delta Analysis: Tracking parameter update magnitudes ($\Delta W$) across attention and MLP layers over training steps (800, 1600, 3000) to inspect mechanistic differences.
Importantly, the authors intentionally avoid auxiliary regularizers (such as dropout) or early stopping to study the raw learning and forgetting dynamics.
Datasets
The study categorizes tasks into two groups:
- Knowledge-Based Benchmarks (also used to quantify forgetting):
- Natural Questions (NQ): Open-domain question answering evaluated as the primary probe for catastrophic forgetting.
- BoolQ: Boolean yes/no reading comprehension.
- GSM8K: Multi-step grade school math reasoning.
- GPQA: Graduate-level "Google-proof" science Q&A.
- Skill-Based Benchmarks (used for task adaptation):
- Syntactic & Morphological Tagging: Universal Part-of-Speech (UPOS), language-specific POS (XPOS), dependency head prediction (Head), morphological feature prediction (FEATS), and lemmatization (LEMMA).
- Reasoning & Planning: Adversarial Natural Language Inference (ANLI), Winograd Schema Challenge (WSC), Blocksworld (PDDL block manipulation), and Logistics (multi-agent transport planning).
Key Insights and Ideas
- SFT causes immediate, catastrophic collapse: Within just a few gradient steps on as few as 16–256 examples of a skill task (UPOS), SFT drives held-out factual knowledge (NQ) accuracy straight to zero. The model begins degenerately annotating instructions for unrelated tasks rather than following them. Lowering the learning rate fails to solve this; it simply impedes learning without preventing forgetting.
- ICL hits a hard capability ceiling: While ICL avoids forgetting entirely, it struggles on complex reasoning and planning. On tasks like Blocksworld and Logistics, ICL yields near-zero accuracy, whereas SFT and LoRA successfully learn the underlying mechanics.
- LoRA needs a "critical mass" of data: Unlike SFT, LoRA cannot fit complex skills from tiny datasets (e.g., 8–16 examples). However, with 64 or more samples, LoRA achieves competitive task accuracy while keeping baseline NQ accuracy stable.
- LoRA is not immune to forgetting: When pushed with larger datasets (512 to 8192 samples) or prolonged training (6k steps), LoRA’s reference task accuracy drops below 20%, showing that parameter-efficient tuning still risks forgetting under extensive fine-tuning.
- Updates localize to high-level layers: LoRA weight deltas ($\Delta W$) are concentrated in late layers (layers 20–31, with isolated spikes at layers 13 and 24), leaving early layers (0–8) virtually untouched. This update pattern stabilizes within 800 steps.
Key Results
- ICL vs. Tuning on Skills: On UPOS, ICL accuracy rises from 83.5% (1 shot) to 89.4% (128 shots). However, on complex structural tasks like syntactic head prediction, ICL maxes out at only 9.6%.
- SFT Overfitting: On syntactic head identification, SFT reaches 100% training accuracy almost immediately, but test accuracy remains near 0% with 16 samples and plateaus at ~20% even with 1024 samples, while destroying general QA abilities.
- LoRA Rank Dynamics: Higher ranks accelerate convergence (ranks 4 and 16 reach ~83% accuracy on UPOS by step 800, whereas rank 1 requires 3,000 steps to reach 83.4%). Diminishing returns occur beyond $r=8$.
Limitations and Future Work
- Single Model Family: The study relies exclusively on Gemma 3; testing across disparate architectures (e.g., mixture-of-experts or non-RMSNorm models) remains open.
- Simplified Training Protocol: Experiments omit standard regularization (e.g., dropout, weight decay, replay buffers, or early stopping), which practitioners often use to curb forgetting.
- Absence of Advanced Prompting: ICL evaluations use standard few-shot prompting, excluding methods like Reinforced ICL or chain-of-thought rationale generation on context lengths exceeding 32k tokens.
Summary updated