A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios

Bernd Bohnet, Rumen Dangovski, Kevin Swersky, Sherry Moore, Kathleen Kenealy, Noah FiedelSubmission to ICLR 2026 · Rejected

TL;DR

Adapting Large Language Models (LLMs) when data is scarce requires navigating a sharp trade-off between skill acquisition and knowledge retention. Comparing Supervised Fine-Tuning (SFT), Low-Rank Adaptation (LoRA), and In-Context Learning (ICL) on Gemma 3 reveals that while SFT learns rapidly from minimal examples, it triggers catastrophic forgetting; ICL completely preserves knowledge but fails at complex reasoning; and LoRA offers the most balanced compromise, provided enough data is available to form an effective low-rank update.

What Problem Does This Paper Address?

When adapting pre-trained LLMs to specialized domains or novel workflows, practitioners typically turn to one of three paradigms: full fine-tuning (SFT), parameter-efficient fine-tuning (PEFT via LoRA), or prompt-based in-context learning (ICL). In data-scarce settings—where only tens or hundreds of labeled examples are available—it remains unclear which method best acquires new capabilities without overwriting the model's pre-trained knowledge and general instruction-following abilities (known as catastrophic forgetting).

Methods

The authors conduct a controlled, head-to-head empirical comparison of standard adaptation methods using Google’s Gemma 3 model within the Kauldron framework:

  • In-Context Learning (ICL): Few-to-many shot prompting (up to 128 shots, fitting within context lengths up to 32k tokens) without gradient updates.
  • Supervised Fine-Tuning (SFT): Full parameter updates across learning rates ($10^{-3}$ to $10^{-4}$) and sample sizes (16 to 1024).
  • Low-Rank Adaptation (LoRA): Low-rank matrix updates evaluated across varying ranks ($r \in \{1, 4, 16\}$) and sample sizes (8 to 8192), using a standard learning rate of $0.005$.
  • Weight Delta Analysis: Tracking parameter update magnitudes ($\Delta W$) across attention and MLP layers over training steps (800, 1600, 3000) to inspect mechanistic differences.

Importantly, the authors intentionally avoid auxiliary regularizers (such as dropout) or early stopping to study the raw learning and forgetting dynamics.

Datasets

The study categorizes tasks into two groups:

  • Knowledge-Based Benchmarks (also used to quantify forgetting):
  • Natural Questions (NQ): Open-domain question answering evaluated as the primary probe for catastrophic forgetting.
  • BoolQ: Boolean yes/no reading comprehension.
  • GSM8K: Multi-step grade school math reasoning.
  • GPQA: Graduate-level "Google-proof" science Q&A.
  • Skill-Based Benchmarks (used for task adaptation):
  • Syntactic & Morphological Tagging: Universal Part-of-Speech (UPOS), language-specific POS (XPOS), dependency head prediction (Head), morphological feature prediction (FEATS), and lemmatization (LEMMA).
  • Reasoning & Planning: Adversarial Natural Language Inference (ANLI), Winograd Schema Challenge (WSC), Blocksworld (PDDL block manipulation), and Logistics (multi-agent transport planning).

Key Insights and Ideas

  • SFT causes immediate, catastrophic collapse: Within just a few gradient steps on as few as 16–256 examples of a skill task (UPOS), SFT drives held-out factual knowledge (NQ) accuracy straight to zero. The model begins degenerately annotating instructions for unrelated tasks rather than following them. Lowering the learning rate fails to solve this; it simply impedes learning without preventing forgetting.
  • ICL hits a hard capability ceiling: While ICL avoids forgetting entirely, it struggles on complex reasoning and planning. On tasks like Blocksworld and Logistics, ICL yields near-zero accuracy, whereas SFT and LoRA successfully learn the underlying mechanics.
  • LoRA needs a "critical mass" of data: Unlike SFT, LoRA cannot fit complex skills from tiny datasets (e.g., 8–16 examples). However, with 64 or more samples, LoRA achieves competitive task accuracy while keeping baseline NQ accuracy stable.
  • LoRA is not immune to forgetting: When pushed with larger datasets (512 to 8192 samples) or prolonged training (6k steps), LoRA’s reference task accuracy drops below 20%, showing that parameter-efficient tuning still risks forgetting under extensive fine-tuning.
  • Updates localize to high-level layers: LoRA weight deltas ($\Delta W$) are concentrated in late layers (layers 20–31, with isolated spikes at layers 13 and 24), leaving early layers (0–8) virtually untouched. This update pattern stabilizes within 800 steps.

Key Results

  • ICL vs. Tuning on Skills: On UPOS, ICL accuracy rises from 83.5% (1 shot) to 89.4% (128 shots). However, on complex structural tasks like syntactic head prediction, ICL maxes out at only 9.6%.
  • SFT Overfitting: On syntactic head identification, SFT reaches 100% training accuracy almost immediately, but test accuracy remains near 0% with 16 samples and plateaus at ~20% even with 1024 samples, while destroying general QA abilities.
  • LoRA Rank Dynamics: Higher ranks accelerate convergence (ranks 4 and 16 reach ~83% accuracy on UPOS by step 800, whereas rank 1 requires 3,000 steps to reach 83.4%). Diminishing returns occur beyond $r=8$.

Limitations and Future Work

  • Single Model Family: The study relies exclusively on Gemma 3; testing across disparate architectures (e.g., mixture-of-experts or non-RMSNorm models) remains open.
  • Simplified Training Protocol: Experiments omit standard regularization (e.g., dropout, weight decay, replay buffers, or early stopping), which practitioners often use to curb forgetting.
  • Absence of Advanced Prompting: ICL evaluations use standard few-shot prompting, excluding methods like Reinforced ICL or chain-of-thought rationale generation on context lengths exceeding 32k tokens.

Summary updated

Abstract

The remarkable capabilities of Large Language Models (LLMs) often need to be tailored for specific applications, requiring the integration of new knowledge or the acquisition of new skills. While full fine-tuning is a powerful adaptation method, it is computationally expensive and can lead to a degradation of general reasoning abilities, a phenomenon known as catastrophic forgetting~\cite{mccloskey1989catastrophic}. A range of alternative techniques exists, each with its own trade-offs. In-Context Learning (ICL) is fast but limited by context length, while Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) offer a middle ground by minimizing parameter changes. However, the challenge of catastrophic forgetting persists, raising questions about the best adaptation strategy for a given task. This paper presents a comparative analysis of Supervised Finetuning (SFT), LoRA, and ICL in data-scarce scenarios. We find that LoRA provides the most effective balance, successfully instilling new skills with minimal impact on the base model's general knowledge. In contrast, while SFT excels at skill acquisition, it is highly susceptible to catastrophic forgetting. ICL is effective for incorporating factual knowledge but struggles with complex skills. Our findings offer a practical framework for selecting an LLM adaptation strategy. We highlight the critical distinction between skill acquisition and knowledge integration, clarify the trade-offs between task-specific performance and the preservation of general capabilities.
BibTeX
@misc{bohnet2025comparative,
  author = {Bernd Bohnet and Rumen Dangovski and Kevin Swersky and Sherry Moore and Kathleen Kenealy and Noah Fiedel},
  title = {{A Comparative Analysis of LLM Adaptation: SFT, LoRA, and ICL in Data-Scarce Scenarios}},
  year = {2025},
  eprint = {2511.00130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.00130},
}
Figure