When adapting a large pretrained foundation model to a downstream task, standard practice dictates appending a randomly initialized classification head and fine-tuning the entire network end-to-end. Intuitively, this joint optimization should allow the model to smoothly adapt its general knowledge to the specific task. However, as demonstrated by Aditi Raghunathan et al. in Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution and supported by Jishnu Mukhoti et al. in Fine-tuning can cripple your foundation model; preserving features may be the solution, full fine-tuning can severely distort or "cripple" the model's high-quality pretrained features. This distortion ultimately leads to poor out-of-distribution (OOD) generalization and representation collapse.
The Gradient Distortion Problem
Why does standard fine-tuning degrade robust pretrained representations? The culprit is the randomly initialized task head. At the start of training, the head's weights are completely unaligned with the target labels, causing it to output massive, highly erroneous losses. When these losses are backpropagated, they generate huge, noisy gradients.
Because the entire network is unfrozen from day one, these chaotic gradients tear through the pretrained backbone. They overwrite the rich, general-purpose feature representations before the head has even learned the basic geometry of the new task. By the time the classification head begins to converge, the underlying feature extractor has already been permanently distorted, losing the broad semantic structure it acquired during large-scale pretraining.
The Two-Stage Solution
To protect the backbone from this initial gradient onslaught, a powerful paradigm has emerged: staged fine-tuning, most prominently implemented as Linear Probing then Fine-Tuning (LP-FT). This approach breaks the adaptation process into two distinct, sequential phases:
- Phase 1: Head Alignment (Linear Probing). The pretrained backbone is frozen, and only the task-specific classification head (or projection layer) is optimized. Because the backbone parameters cannot change, the head is forced to find a configuration that works with the existing feature space. This phase continues until the head stabilizes and achieves near-optimal alignment with the pretrained representations.
- Phase 2: Full Adaptation (Fine-Tuning). The entire network is unfrozen, and full-parameter updates are performed. Crucially, because the classification head is already well-aligned with the task, the gradients generated at the start of this phase are small, stable, and precise. Instead of destroying the feature space, they gently warp it to fit the nuances of the downstream domain.
In their analysis of catastrophic forgetting, Vinay Venkatesh Ramasesh et al. in Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics show that this "head-first" training acts as a protective buffer, preventing high-magnitude gradients from erasing pre-existing representations in the network's deeper layers. Similarly, Jaehyun Pahk et al. leverage this mechanism for dense prediction tasks in Decoupled Finetuning for Domain Generalizable Semantic Segmentation, demonstrating that freezing the encoder while training a randomly initialized decoder prevents early feature distortion and preserves domain generalizability.
Geometric and Optimization Intuition
Why does this sequence matter so much for the final model's performance? The answer lies in the geometry of the loss landscape.
When a model undergoes full fine-tuning from a random head initialization, the massive early gradients push the model's parameters far away from their pretrained starting point, often dropping them into suboptimal, highly specialized local minima. Jeevesh Juneja et al. demonstrate in Linear Connectivity Reveals Generalization Strategies that models can fall into entirely different geometric basins depending on their optimization trajectory, which directly dictates their generalization strategies (such as relying on simple heuristics versus understanding complex syntax). By optimizing the head independently first, staged fine-tuning anchors the optimization trajectory, forcing the model into a consistent, predictable basin that retains generalizable properties.
Furthermore, Donghoon Kim et al. observe in Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models that establishing task-specific context before updating weights prevents representation collapse in large multi-modal architectures, keeping the internal knowledge structures intact.
Robustness Benefits Across Domains
The benefits of staged fine-tuning extend far beyond standard classification accuracy; it serves as a core mechanism for building robust, resilient models across diverse settings:
- Noisy Labels: In datasets with high label noise, standard fine-tuning quickly memorizes errors due to aggressive gradient updates. Sumyeong Ahn et al. show in Fine-tuning Pre-trained Models for Robustness under Noisy Labels that sequencing linear probing before full fine-tuning acts as a protective shield, allowing the model to identify clean data subsets and task-relevant features before updating the backbone, preventing noise memorization.
- Federated Learning: In decentralized environments, local fine-tuning can cause "federated feature distortion," where models overfit to local user distributions. Hrad Ghoukasian et al. demonstrate in A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learning that a two-phase LP-FT strategy stabilizes the local classification heads, preserving the integrity of the shared global representations.
- Self-Supervised Learning (SSL): Representations learned via non-contrastive SSL are highly sensitive to corruption. Jaejin Cho et al. find in Non-Contrastive Self-Supervised Learning for Utterance-Level Information Extraction From Speech that a two-stage fine-tuning process protects speech representations from being ruined by the noisy gradients of randomly initialized task layers.
- Unsupervised Domain Adaptation: When adapting foundation models to new domains without labels, Jesimon Barreto et al. utilize this concept in VESSA: Video-based objEct-centric Self-Supervised Adaptation for Visual Foundation Models via staged projection head alignment, preventing the gradients of a new head from degrading pretrained weights during the initial phases of unsupervised tuning.
By recognizing that the order of parameter updates matters just as much as the data itself, staged fine-tuning provides a reliable blueprint for adapting foundation models without sacrificing their inherent intelligence.