Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning

Dongze Lian, Daquan Zhou, Jiashi Feng, Xinchao Wang

TL;DR

SSF (Scale and Shift Features) is a simple yet powerful parameter-efficient fine-tuning method that outperforms full fine-tuning by modulating deep features with learnable scale and shift parameters. Because these transformations are linear, they can be merged back into the original model weights via re-parameterization, achieving state-of-the-art performance with zero additional inference overhead.

What Problem Does This Paper Address?

As deep learning models grow to billions of parameters, fine-tuning them for specific downstream tasks becomes a bottleneck. Full fine-tuning (updating all weights) is computationally expensive, requires massive storage for each task-specific model, and often leads to over-fitting on small datasets. Conversely, linear probing (tuning only the final classification head) is efficient but usually results in a significant drop in accuracy.

Recent "parameter-efficient" alternatives like Adapters or Visual Prompt Tuning (VPT) have bridged this gap, but they come with their own baggage: they often introduce task-dependent architectures or additional "tokens" that increase computational cost (FLOPs) and latency during inference. The authors of this paper sought a "Goldilocks" solution: a method that is as accurate as full fine-tuning, as efficient as linear probing, and introduces no extra cost at runtime.

Methods

The core proposal is SSF (Scale and Shift Features). Instead of adding complex sub-networks or prompts, the authors insert a simple linear transformation after every operation (like Multi-Head Self-Attention, MLP blocks, or Layer Normalization) in a pre-trained model.

Mathematically, for a feature $x$, the output $y$ is calculated as: $$y = \gamma \odot x + \beta$$ where $\gamma$ and $\beta$ are learnable scale and shift vectors.

The SSF training and inference pipeline. Note how SSF-ADA modules are merged for inference.

The "magic" happens during the transition from training to deployment. Because scaling and shifting are linear operations, they can be mathematically "absorbed" into the weights and biases of the preceding layer. This process, known as re-parameterization, means that the final deployed model has the exact same architecture and speed as the original pre-trained backbone, just with updated weight values.

Datasets

The authors conducted an extensive evaluation across 26 image classification datasets:

  • FGVC (Fine-Grained Visual Classification): 5 datasets including CUB-200-2011 (birds) and Stanford Cars.
  • VTAB-1k (Visual Task Adaptation Benchmark): A challenging suite of 19 diverse tasks ranging from natural images to medical and satellite imagery.
  • General Classification: Large-scale benchmarks like CIFAR-100 and ImageNet-1K.
  • Robustness & OOD: ImageNet-A, ImageNet-R, and ImageNet-C to test how well the model handles adversarial examples and corruptions.

Key Insights and Ideas

  • Distribution Matching: The authors argue that the primary hurdle in transfer learning is the "distribution mismatch" between the upstream (pre-training) and downstream (target) data. SSF acts as a lightweight bridge, modulating features to fall into a discriminative space for the new task.
  • Occam’s Razor: SSF proves that you don't need complex architectural changes to achieve high performance. Tuning just two vectors per layer is often more effective than adding non-linear adapter modules.
  • Zero-Overhead Inference: Unlike VPT or Adapters, which increase test-time latency, SSF is "free" at runtime. This makes it ideal for deployment on edge devices like mobile phones.
  • Model Agnostic: While tested heavily on Vision Transformers (ViT), the authors demonstrate that SSF works equally well across CNNs (ConvNeXt) and MLP-based architectures (AS-MLP).

Key Results

SSF consistently outperformed both full fine-tuning and existing parameter-efficient methods while using a fraction of the tunable parameters (typically ~0.3M vs ~86M for a ViT-B backbone).

  • Superior Accuracy: On the FGVC benchmark, SSF achieved 90.72% Top-1 accuracy, surpassing full fine-tuning (88.54%) and VPT-Deep (89.11%).
  • Massive Gains on Small Data: On the VTAB-1k benchmark, SSF reached 73.10%, a staggering 11.48% improvement over full fine-tuning.
  • Efficiency: As shown in Figure 1, SSF sits at the top-left of the accuracy-vs-parameters curve, representing the best trade-off in the field.
Figure 1: Performance comparisons of seven fine- tuning methods with a pre-trained ViT-B/16 model on the FGVC dataset and VTAB-1k benchmark. Our SSF (red dots) achieves state-of-the-art per- formance only with about 0.3M average learnable parameters.
  • Inference Speed: Benchmarks confirmed that SSF has the lowest test time among all high-performing tuning methods, matching the speed of the original frozen backbone.
Test time comparison showing SSF's zero-overhead advantage.

Limitations and Future Work

While SSF is highly effective for classification, the authors note a few areas for improvement:

  • Dense Prediction Tasks: In complex tasks like object detection and semantic segmentation, SSF outperforms linear probing but still lags slightly behind full fine-tuning. The authors suggest that these tasks might require more than just feature modulation.
  • Task Correlations: Currently, SSF treats every task independently. Future work could explore how to correlate multiple tasks during fine-tuning to further boost performance.
  • Backbone Bias: Like all fine-tuning methods, SSF is limited by the quality of the pre-trained backbone. If the upstream model was trained on biased or "illegal" data, SSF will likely inherit those flaws.

Summary updated

Abstract

Existing fine-tuning methods either tune all parameters of the pre-trained model (full fine-tuning), which is not efficient, or only tune the last linear layer (linear probing), which suffers a significant accuracy drop compared to the full fine-tuning. In this paper, we propose a new parameter-efficient fine-tuning method termed as SSF, representing that researchers only need to Scale and Shift the deep Features extracted by a pre-trained model to catch up with the performance of full fine-tuning. In this way, SSF also surprisingly outperforms other parameter-efficient fine-tuning approaches even with a smaller number of tunable parameters. Furthermore, different from some existing parameter-efficient fine-tuning methods (e.g., Adapter or VPT) that introduce the extra parameters and computational cost in the training and inference stages, SSF only adds learnable parameters during the training stage, and these additional parameters can be merged into the original pre-trained model weights via re-parameterization in the inference phase. With the proposed SSF, our model obtains 2.46% (90.72% vs. 88.54%) and 11.48% (73.10% vs. 65.57%) performance improvement on FGVC and VTAB-1k in terms of Top-1 accuracy compared to the full fine-tuning but only fine-tuning about 0.3M parameters. We also conduct amounts of experiments in various model families (CNNs, Transformers, and MLPs) and datasets. Results on 26 image classification datasets in total and 3 robustness & out-of-distribution datasets show the effectiveness of SSF. Code is available at https://github.com/dongzelian/SSF.
BibTeX
@inproceedings{lian2022scaling,
  author = {Dongze Lian and Daquan Zhou and Jiashi Feng and Xinchao Wang},
  title = {{Scaling \& Shifting Your Features: A New Baseline for Efficient Model Tuning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=dTZuTZUf00},
}

Loading related content...

Figure