Staged Adaptation to Preserve Pretrained Features

123 concepts95 papers

When adapting a large pretrained foundation model to a downstream task, standard practice dictates appending a randomly initialized classification head and fine-tuning the entire network end-to-end. Intuitively, this joint optimization should allow the model to smoothly adapt its general knowledge to the specific task. However, as demonstrated by Aditi Raghunathan et al. in Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution and supported by Jishnu Mukhoti et al. in Fine-tuning can cripple your foundation model; preserving features may be the solution, full fine-tuning can severely distort or "cripple" the model's high-quality pretrained features. This distortion ultimately leads to poor out-of-distribution (OOD) generalization and representation collapse.

The Gradient Distortion Problem

Why does standard fine-tuning degrade robust pretrained representations? The culprit is the randomly initialized task head. At the start of training, the head's weights are completely unaligned with the target labels, causing it to output massive, highly erroneous losses. When these losses are backpropagated, they generate huge, noisy gradients.

Because the entire network is unfrozen from day one, these chaotic gradients tear through the pretrained backbone. They overwrite the rich, general-purpose feature representations before the head has even learned the basic geometry of the new task. By the time the classification head begins to converge, the underlying feature extractor has already been permanently distorted, losing the broad semantic structure it acquired during large-scale pretraining.

The Two-Stage Solution

To protect the backbone from this initial gradient onslaught, a powerful paradigm has emerged: staged fine-tuning, most prominently implemented as Linear Probing then Fine-Tuning (LP-FT). This approach breaks the adaptation process into two distinct, sequential phases:

  1. Phase 1: Head Alignment (Linear Probing). The pretrained backbone is frozen, and only the task-specific classification head (or projection layer) is optimized. Because the backbone parameters cannot change, the head is forced to find a configuration that works with the existing feature space. This phase continues until the head stabilizes and achieves near-optimal alignment with the pretrained representations.
  2. Phase 2: Full Adaptation (Fine-Tuning). The entire network is unfrozen, and full-parameter updates are performed. Crucially, because the classification head is already well-aligned with the task, the gradients generated at the start of this phase are small, stable, and precise. Instead of destroying the feature space, they gently warp it to fit the nuances of the downstream domain.

In their analysis of catastrophic forgetting, Vinay Venkatesh Ramasesh et al. in Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics show that this "head-first" training acts as a protective buffer, preventing high-magnitude gradients from erasing pre-existing representations in the network's deeper layers. Similarly, Jaehyun Pahk et al. leverage this mechanism for dense prediction tasks in Decoupled Finetuning for Domain Generalizable Semantic Segmentation, demonstrating that freezing the encoder while training a randomly initialized decoder prevents early feature distortion and preserves domain generalizability.

Geometric and Optimization Intuition

Why does this sequence matter so much for the final model's performance? The answer lies in the geometry of the loss landscape.

When a model undergoes full fine-tuning from a random head initialization, the massive early gradients push the model's parameters far away from their pretrained starting point, often dropping them into suboptimal, highly specialized local minima. Jeevesh Juneja et al. demonstrate in Linear Connectivity Reveals Generalization Strategies that models can fall into entirely different geometric basins depending on their optimization trajectory, which directly dictates their generalization strategies (such as relying on simple heuristics versus understanding complex syntax). By optimizing the head independently first, staged fine-tuning anchors the optimization trajectory, forcing the model into a consistent, predictable basin that retains generalizable properties.

Furthermore, Donghoon Kim et al. observe in Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models that establishing task-specific context before updating weights prevents representation collapse in large multi-modal architectures, keeping the internal knowledge structures intact.

Robustness Benefits Across Domains

The benefits of staged fine-tuning extend far beyond standard classification accuracy; it serves as a core mechanism for building robust, resilient models across diverse settings:

By recognizing that the order of parameter updates matters just as much as the data itself, staged fine-tuning provides a reliable blueprint for adapting foundation models without sacrificing their inherent intelligence.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_staged_fine_tuning_6dd8ef20,
  title = {staged fine-tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/staged-fine-tuning/rdr_6dd8ef20ce2d4491b624ae61a71bbbf7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_staged_fine_tuning_6dd8ef20_v1,
  title = {staged fine-tuning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/staged-fine-tuning/rdr_6dd8ef20ce2d4491b624ae61a71bbbf7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake