The standard paradigm in modern deep learning is intuitive: pre-train a model on a massive, diverse dataset to learn general features, and then fine-tune its weights on a specific target task. However, full fine-tuning suffers from a hidden pathology: destructive interference. When we update an entire network on a downstream task, the optimization process often overwrites and erases the rich, general-purpose representations learned during pre-training.
To prevent this degradation, a growing body of research advocates for a simpler, more conservative strategy: freezing the pre-trained feature extractor and training only the task-specific downstream layers.
The Optimization-Generalization Dilemma
Why does full fine-tuning destroy valuable features? The answer lies in how neural networks optimize for a local objective. As Lopez-Paz et al. demonstrate, optimization and out-of-distribution (OoD) generalization are frequently at odds. When a network is allowed to update its entire backbone on a new dataset, it searches for the path of least resistance to minimize training loss. This often leads to "shortcut learning," where the model discards robust, complex features in favor of simpler, spurious correlations unique to the training distribution.
By fixing the parameters of a pre-trained feature extractor, we force the model to draw from a pre-existing "palette" of diverse features. This constraint prevents the optimization process from warping the underlying feature space to fit shortcuts, thereby preserving the model's ability to generalize to unseen distributions. Similarly, Ye et al. show that locking a subset of features derived from label-agnostic methods prevents the model from discarding noisy but ultimately relevant information, enhancing robustness against distribution shifts.
A Protective Buffer Against Noisy Data
The vulnerability of full fine-tuning is severely magnified when dealing with imperfect or messy data. In scenarios with high label noise or weak supervision, task-specific gradients carry erroneous, highly destructive signals.
[Noisy Downstream Data] ──> Gradients ──> [Classification Head] ──X──> [Frozen Backbone]
(Features Preserved)
Freezing the backbone acts as a protective shield against these corrupted gradients:
- Label Noise: Ahn et al. demonstrate that freezing the feature extractor while training only the classification head serves as a buffer, preventing high-quality representations from being distorted by incorrect labels during initial adaptation. Furthermore, Xu et al. find that training a simple linear classifier on top of frozen self-supervised representations maintains the integrity of learned feature clusters. Full fine-tuning on the same noisy data causes the model to warp its internal structures to memorize the noise.
- Weak Supervision: In weakly supervised object detection (WSOD), where models must learn to localize objects using only image-level labels, early training signals are notoriously unreliable. Shen et al. show that freezing the initial stages of a pre-trained backbone prevents erroneous localization gradients from destroying foundational visual knowledge.
Preserving Spatial Hierarchies and Semantic Discrimination
Destructive interference also occurs when adapting models across different task types, such as moving from image classification to object detection or spatial mapping.
When a model undergoes long training schedules on a specialized downstream task, the optimization process can erase general-purpose spatial hierarchies. Vasconcelos et al. show that keeping backbone weights static prevents this erasure, resulting in significantly better generalization—particularly for rare objects in long-tail distributions.
This preservation is equally vital during unsupervised pretext training. For instance, Dai et al. observe that allowing a backbone to update during a spatial localization pretext task inadvertently degrades high-level semantic classification features. Freezing the feature extraction backbone forces the downstream transformer modules to learn spatial mapping without eroding the backbone's semantic discrimination.
Open-World and Synthetic Generalization
When a model must generalize to novel classes or entirely new domains, frozen features provide a stable, uncorrupted anchor:
- Open-World Tasks: In few-shot object counting, You et al. note that fine-tuning a feature extractor impairs generalization to unseen categories because the broad semantic representations required for open-world tasks are lost.
- Synthetic-to-Real Shifts: When training on synthetic data, Benigmim et al. show that freezing the backbone prevents the model from overfitting to the specific visual artifacts of the synthetic domain, preserving its real-world generalization capabilities.
- Small Datasets & Sample Efficiency: Xompero et al. highlight that freezing early layers maintains the generalization benefits of adversarial pre-training when adapting to small datasets. In reinforcement learning, Zhang et al. show that maintaining fixed representations yields higher sample efficiency because task-specific gradients are prevented from degrading general structural knowledge of environment dynamics. Similarly, Naimi et al. find that freezing initial spatial layers after self-supervised pre-training preserves robust, low-level features that outperform fully fine-tuned alternatives in skeleton action recognition.
Alternative Paradigms of Feature Preservation
Freezing does not have to be a binary choice between static representations and full fine-tuning. Several soft preservation strategies have emerged to balance adaptation with stability:
- Gradient Masking: Kim et al. utilize selective parameter updates and gradient masking to ensure that a fine-tuned model's internal representations maintain high similarity to its original pre-trained state, shielding foundational knowledge during adversarial training.
- Stochastic Cross-Attention: Seo et al. introduce a mechanism that intermittently queries a frozen, pre-trained reference model via cross-attention during fine-tuning. This allows the target model to selectively consult pre-trained knowledge, mitigating feature distortion while still allowing adaptation.
- Early Pre-training Termination: Feature preservation can also be managed by altering the pre-training schedule itself. Schneider demonstrates that terminating pre-training at the point of minimum reconstruction loss—rather than peak source-task accuracy—prevents the model from entering an over-specialization phase where it discards the very general-purpose features needed for downstream transfer.
Key Foundational Work
UP-DETR: Unsupervised Pre-Training for Object Detection With Transformers (2021) — contributed 3 concept elements to this direction.
Enabling Deep Residual Networks for Weakly Supervised Object Detection (2020) — contributed 2 concept elements to this direction.
Improving filling level classification with adversarial training (2021) — contributed 1 concept elements to this direction.