Sequential Linear Probing and Fine-Tuning for Robust Adaptation

This direction is split into other directions: staged fine-tuning. Its last version stays readable and citable here.

Version 1 of 1102 papers125 conceptsfrozen 2026-10-04

When we adapt a massive pre-trained model to a new task, the standard protocol is "fine-tuning": we attach a randomly initialized classification head and update every weight in the network. While this is the industry standard, researchers have recently discovered that this process is surprisingly "violent" to the model’s internal knowledge.

The core insight emerging across dozens of papers is that standard fine-tuning often destroys the very features that made the pre-trained model valuable in the first place. To fix this, a consensus is forming around a two-stage strategy: Linear Probing followed by Fine-Tuning (LP-FT).

The Problem: Feature Distortion

The intuition behind the failure of standard fine-tuning is simple: at the start of training, the new classification head is random. Because it knows nothing about the task, it produces massive errors. These errors generate huge gradients that are backpropagated through the entire network.

As Aditi Raghunathan, Ananya Kumar, and Percy Liang demonstrate in Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution, these initial "noisy" gradients act like a sledgehammer, distorting the high-quality representations in the backbone to fit the needs of a poorly initialized head. This leads to a "robustness gap" where the model performs well on the specific training data but fails miserably on "out-of-distribution" (OOD) data—scenarios where the test data differs slightly from the training set, as explored in the GLUE-X benchmark by Linyi Yang and colleagues.

The Solution: A Two-Stage Handshake

The LP-FT strategy acts as a "warm-up" or a handshake between the old knowledge and the new task. It breaks the adaptation into two distinct phases:

  1. Linear Probing (LP): You freeze the pre-trained backbone and only train the new classification head. This forces the head to "find" the relevant features already present in the model.
  2. Fine-Tuning (FT): Once the head is stable and aligned with the feature space, you unfreeze the entire network and fine-tune all parameters.

By the time the second stage begins, the head is already producing reasonable predictions. The gradients sent back to the backbone are now "surgical" rather than "destructive," allowing the model to refine its features without losing its general-purpose foundation. Jihed Chelbi and José Dolz showed this is particularly effective in medical imaging, where preserving domain-specific features is critical for detecting rare conditions like hypertensive retinopathy.

Stability Across Domains

This two-stage intuition is appearing in almost every subfield of machine learning:

  • Remote Sensing: Anthony Fuller and Koreen Millard found that SatViT-V2 handles global biome shifts better when the linear head is optimized before full-model fine-tuning.
  • Federated Learning: In decentralized settings, Hrad Ghoukasian and Xiaoxiao Li argue that LP-FT prevents "feature distortion" caused by the diverse, heterogeneous data held by different users.
  • Continual Learning: To prevent "catastrophic forgetting," Jinghuan Shang and Niranjan Balasubramanian use a similar approach in Global Alignment, ensuring class vectors are well-positioned before the representation space is adapted.
  • Natural Language Processing: Even in translation, Aniruddha Roy and Ayush Maheshwari found that training a task-specific decoder first while keeping the encoder frozen prevents the degradation of pre-trained multilingual features.

Why Timing Matters More Than Complexity

Perhaps the most surprising finding, highlighted by Massimiliano Mancini and Elisa Ricci in Rethinking Few-Shot Adaptation, is that the sequence of training is often more important than the architecture of the adapter. Their 2SFS method shows that knowing when to stop tuning the backbone and start training the classifier can outperform much more complex, parameter-heavy adaptation techniques.

Whether it is Elastic Neural Networks using auxiliary branches or Vision Transformers using patch-slimming modules, the recurring lesson is the same: stabilize the new components first. By treating the pre-trained backbone as a fragile library of knowledge rather than a blank slate, we can build models that are not just accurate on their training sets, but robust in the real world.

Papers in this version

Showing 60 of 102 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sequential_linear_probing_and_fine_tunin_04577fac,
  title = {Sequential Linear Probing and Fine-Tuning for Robust Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-linear-probing-and-fine-tuning-for-robust-adaptation/rdr_04577fac56c24ed59ae78d3d4267f898},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequential_linear_probing_and_fine_tunin_04577fac_v1,
  title = {Sequential Linear Probing and Fine-Tuning for Robust Adaptation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequential-linear-probing-and-fine-tuning-for-robust-adaptation/rdr_04577fac56c24ed59ae78d3d4267f898/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}