Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning

Dongze Lian, Daquan Zhou, Jiashi Feng, Xinchao Wang

TL;DR

SSF (Scale and Shift Features) is a parameter-efficient fine-tuning method that outperforms full fine-tuning by adjusting only a tiny fraction of parameters (~0.3M). By applying simple linear transformations to deep features, SSF achieves state-of-the-art accuracy on downstream tasks while allowing all new parameters to be "folded" back into the original model. This results in zero additional computational overhead during inference.

What Problem Does This Paper Address?

As deep learning models grow to billions of parameters (e.g., ViT-G, CoAtNet), the standard practice of full fine-tuning—updating every single weight for every new task—has become unsustainable. It requires massive storage (a full copy of the model per task) and risks overfitting on smaller datasets.

Existing alternatives have significant drawbacks:

  1. Linear Probing: Only tuning the last layer is efficient but usually results in a major accuracy drop.
  2. Adapters and Visual Prompt Tuning (VPT): These "Parameter-Efficient Fine-Tuning" (PEFT) methods insert extra layers or input tokens. While they reduce the number of trainable parameters, they increase the computational cost (FLOPs) and latency during inference, making them difficult to deploy on edge devices like mobile phones.

The authors seek a "Goldilocks" solution: a method that is as accurate as full fine-tuning, as efficient as linear probing, and adds zero cost to the final deployed model.

Methods

The proposed method, SSF, is elegantly simple. Instead of adding complex sub-networks, it introduces two learnable vectors for every operation (like Multi-Head Attention or MLP blocks) in a pre-trained model:

  • Scale ($\gamma$): A vector that performs element-wise multiplication on the features.
  • Shift ($\beta$): A vector that is added to the features.

The transformation is defined as: $y = \gamma \odot x + \beta$.

SSF Architecture and Folding Trick

The "Folding" Trick (Re-parameterization)

The brilliance of SSF lies in its linearity. Because the scale and shift operations are linear, they can be mathematically merged into the weights and biases of the preceding layer once training is finished.

  • During Training: You only update $\gamma$ and $\beta$ (keeping the backbone frozen).
  • During Inference: You "absorb" these parameters into the original weights.

This means the final deployed model has the exact same architecture and speed as the original pre-trained model, but with "tuned" weights optimized for the new task.

Datasets

The authors evaluated SSF across a massive suite of 26 image classification datasets:

  • FGVC (Fine-Grained Visual Classification): 5 datasets including CUB-200-2011, Oxford Flowers, and Stanford Cars.
  • VTAB-1k: A benchmark of 19 diverse tasks ranging from natural images to specialized medical and satellite imagery.
  • General Classification: CIFAR-100 and ImageNet-1K.
  • Robustness & OOD: ImageNet-A, ImageNet-R, and ImageNet-C to test how well the model handles corrupted or out-of-distribution data.

Key Insights and Ideas

  • Distribution Matching is Key: The authors argue that the main challenge in transfer learning is the "distribution mismatch" between the upstream (pre-training) and downstream (target) data. SSF acts as a lightweight bridge that modulates feature distributions (variance and mean) to fit the new task.
  • Occam’s Razor in PEFT: While other methods use non-linear adapters or complex prompting, SSF shows that simple linear modulation is often more effective. It achieves better results with fewer parameters.
  • Universal Applicability: Unlike VPT, which is mostly tied to Transformers, SSF is "plug-and-play." The authors successfully applied it to CNNs (ConvNeXt), Transformers (ViT, Swin), and even MLP-based architectures (AS-MLP).
  • Better Robustness: Surprisingly, SSF-tuned models often showed better robustness to image corruptions than full fine-tuning. This is likely because freezing the backbone preserves the general features learned on massive datasets, preventing the model from "forgetting" general visual knowledge.

Key Results

Performance Comparison on FGVC and VTAB-1k
  • Superior Accuracy: On the VTAB-1k benchmark, SSF achieved 73.10% Top-1 accuracy, significantly outperforming full fine-tuning (65.57%) and VPT-Deep (69.43%).
  • Efficiency: SSF achieved these results by tuning only 0.24M parameters for VTAB, compared to the 85.8M parameters required for full fine-tuning.
  • Zero Overhead: Unlike Adapters or VPT, which increase inference time and memory usage, SSF's inference cost is identical to the original backbone.
  • State-of-the-Art on FGVC: SSF reached 90.72% average accuracy across five fine-grained datasets, beating the previous best PEFT methods.

Limitations and Future Work

  • Large-Scale Data Gap: While SSF dominates on small-to-medium datasets (FGVC, VTAB), full fine-tuning still holds a slight edge on massive datasets like ImageNet-1K (83.58% vs 83.10%). When data is abundant, the constraints of PEFT can become a slight bottleneck.
  • Manual Placement: The current work inserts SSF after every operation. The authors suggest that using Neural Architecture Search (NAS) to find optimal insertion points could further reduce parameter counts, though they opted for the simpler "insert everywhere" approach for this baseline.
  • Task Independence: While SSF is parameter-efficient, it still requires a set of scale/shift parameters per task. Future work could explore how to share these parameters across related tasks.

Summary updated

Abstract

Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, we propose a new parameter-efficient fine-tuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine-tuning. In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters. Furthermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MLPs) and datasets. Results on 26 image classification datasets in total and 3 robustness & out-of-distribution datasets show the effectiveness of SSF. Code is available at https://github.com/dongzelian/SSF.
BibTeX
@inproceedings{lian2022scaling,
  author = {Dongze Lian and Daquan Zhou and Jiashi Feng and Xinchao Wang},
  title = {{Scaling \& Shifting Your Features: A New Baseline for Efficient Model Tuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=dTZuTZUf00},
}
Figure