Abstract
Existing fine-tuning methods typically face a trade-off between efficiency and performance. Full fine-tuning updates all model parameters, which is resource-intensive, while linear probing—tuning only the final classification layer—often results in significant accuracy drops. In this paper, we propose SSF (Scale and Shift Features), a parameter-efficient fine-tuning method that modulates deep features extracted by a pre-trained model. By applying simple linear transformations, SSF matches or exceeds the performance of full fine-tuning while updating significantly fewer parameters. Notably, SSF introduces zero additional computational cost during inference through a re-parameterization trick. Our experiments across 26 image classification datasets and various architectures (CNNs, Transformers, and MLPs) demonstrate that SSF achieves a 2.46% improvement on FGVC and an 11.48% improvement on VTAB-1k compared to full fine-tuning, while tuning only approximately 0.3M parameters.
Introduction
As deep learning models continue to scale in size—exemplified by architectures like ViT-G and CoAtNet—the traditional paradigm of "full fine-tuning" has become increasingly impractical. Storing a complete set of parameters for every downstream task is storage-intensive and prone to overfitting on smaller datasets.
To address this, researchers have explored Parameter-Efficient Fine-Tuning (PEFT). Current PEFT strategies generally fall into two categories:
- Linear Probing: Tuning only the final head. This is efficient but lacks the capacity to adapt the backbone's internal representations to new domains.
- Structural Adapters/Prompts: Methods like Adapters or Visual Prompt Tuning (VPT) insert new modules or parameters into the architecture. While effective, these often increase inference latency and memory consumption because the model architecture is fundamentally altered.
SSF provides a "Goldilocks" solution: it achieves the high performance of full fine-tuning with the low parameter count of PEFT, all while maintaining the original model's inference speed.
Methodology
The SSF Operation
The core of SSF is a simple linear transformation applied to the features produced by every operation within a pre-trained model (e.g., after Multi-head Self-Attention, MLPs, or Layer Normalization). For any feature $x$, the transformed output $y$ is defined as:
$$y = \gamma \odot x + \beta$$
In this equation, $\gamma$ represents a scale factor (modulating variance) and $\beta$ represents a shift factor (modulating the mean). The symbol $\odot$ denotes the element-wise product. By adjusting these two vectors, the model can align the distribution of pre-trained features with the requirements of the target task.
Re-parameterization for Inference Efficiency
The most significant advantage of SSF is its ability to be merged back into the original model weights. Because the transformation is linear, the learned $\gamma$ and $\beta$ can be mathematically integrated into the weights ($w$) and biases ($b$) of the preceding layer during the inference phase.
A standard operation $w * t + b$ followed by SSF becomes: $$y = (\gamma \odot w) * t + (\gamma \odot b + \beta)$$
This means that at test time, the model architecture remains identical to the original pre-trained version. There are no extra layers to compute, resulting in zero additional latency or FLOPs compared to the backbone model.
Experimental Results
Performance Benchmarks
We evaluated SSF across a comprehensive suite of 26 datasets, including the Fine-Grained Visual Classification (FGVC) tasks and the VTAB-1k benchmark.
- FGVC: SSF achieved a Top-1 accuracy of 90.72%, outperforming full fine-tuning (88.54%) and VPT-Deep (89.11%).
- VTAB-1k: On this diverse benchmark (19 tasks), SSF showed a massive gain, reaching 73.10% compared to the 65.57% achieved by full fine-tuning.
- Parameter Efficiency: For a ViT-B backbone, SSF requires tuning only 0.3M parameters, a tiny fraction of the 86M parameters required for full fine-tuning.
Robustness and Generalization
Beyond standard accuracy, SSF demonstrates superior performance on out-of-distribution (OOD) and corrupted data. Testing on ImageNet-A, ImageNet-R, and ImageNet-C showed that SSF generalizes better than full fine-tuning. This suggests that by only scaling and shifting features rather than overhauling the entire weight matrix, SSF preserves the robust representations learned during large-scale pre-training.
Architectural Versatility
Unlike many PEFT methods tailored specifically for Transformers, SSF is architecture-agnostic. We successfully applied it to:
- CNNs: ConvNeXt
- Transformers: ViT, Swin
- MLPs: AS-MLP
In all cases, SSF consistently improved performance over baseline tuning methods, proving its utility as a general-purpose fine-tuning tool.
Conclusion
SSF (Scale and Shift Features) offers a simple yet highly effective baseline for model adaptation. By focusing on distribution alignment through linear transformations, it bypasses the need for complex, non-linear adapters. Its ability to achieve state-of-the-art results while remaining re-parameterizable makes it an ideal choice for deploying large-scale models in resource-constrained or latency-sensitive environments. While full fine-tuning still holds a minor edge in extremely large-data regimes (like ImageNet-1K), SSF is the clear winner for the vast majority of transfer learning tasks.
Summary updated