Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks

Jeffrey O. Zhang, Alexander Sax, Amir Zamir, Leonidas J. Guibas, Jitendra Malik

Introduction

The rapid advancement of deep learning has made pre-trained models a cornerstone of modern AI development. However, adapting these models to new, specialized tasks remains a challenge. Traditionally, practitioners choose between two primary strategies: fine-tuning and feature extraction (fixed features).

Fine-tuning involves updating all weights of a pre-trained model on a new dataset. While flexible, it is prone to "catastrophic forgetting," where the model loses its original capabilities, and overfitting, especially when the target dataset is small. Conversely, feature extraction keeps the pre-trained model frozen and only trains a new output layer. While this preserves the original knowledge and is computationally efficient, it is inherently "rigid." If the pre-trained model has discarded features essential for the new task, the model's performance is fundamentally capped.

This paper introduces Side-Tuning, an additive approach that seeks a middle ground. By training a lightweight "side" network alongside a frozen base model, side-tuning offers the flexibility to learn new information without compromising the stability of the original pre-trained representations.

Methodology

The core architecture of Side-Tuning is elegantly simple. Rather than modifying the internal weights of a pre-trained base model ($B$), the method introduces a separate, lightweight side network ($S$).

Comparison of network adaptation strategies: Fixed Features (frozen base), Fine-Tuning (updated base), and Side-Tuning (frozen base with additive side network).

The process follows three main steps:

  1. Frozen Base Model: The pre-trained weights of $B$ are kept strictly frozen, ensuring zero degradation of the original knowledge.
  2. Additive Side Network: A small side network $S$ is initialized (either from scratch or via distillation) and trained specifically for the target task.
  3. Alpha-Blending: The final representation $R(x)$ for an input $x$ is calculated as a weighted combination of the outputs from both networks: $$R(x) = \alpha B(x) + (1 - \alpha) S(x)$$

The blending parameter $\alpha$ serves as a crucial hyperparameter. It can be fixed, learned during training, or adjusted via a curriculum (e.g., starting high to leverage the base model and decreasing to allow the side network to specialize). Because the base model is treated as a "black box," side-tuning is architecture-agnostic; it can be applied to any model that produces a vector representation, regardless of whether it is a ResNet, a Transformer, or even a non-neural model.

Experimental Evaluation

The authors evaluated Side-Tuning across diverse domains, including computer vision, natural language processing (NLP), and reinforcement learning (RL).

Benchmarks and Datasets

  • iTaskonomy: A significant contribution of this work is the introduction of iTaskonomy, an incremental learning benchmark based on the Taskonomy dataset. It features 12 distinct vision tasks (e.g., edge detection, object classification, surface normals) performed on a shared set of images, providing a rigorous test for forward transfer and forgetting.
  • iCIFAR: An incremental version of CIFAR-100 split into 10 sequential tasks.
  • SQuAD v2: A standard NLP benchmark for question-answering.
  • Habitat: A simulation environment for evaluating visual navigation in RL and imitation learning contexts.

Key Results

Side-tuning consistently demonstrated superior performance and efficiency:

  • Incremental Learning Superiority: On iTaskonomy, side-tuning achieved the best average rank (1.33) among competing methods, outperforming Progressive Neural Networks (2.42) and Elastic Weight Consolidation (EWC, 5.25).
  • Parameter Efficiency: In the iTaskonomy experiments, side-tuning required only 11M parameters to cover all tasks, significantly less than the 24.6M required by EWC or Parameter Superposition (PSP).
  • Cross-Domain Robustness: In NLP tasks using BERT, side-tuning matched the performance of full fine-tuning. In RL navigation, it effectively bridged the gap between fixed features and training from scratch, proving particularly potent when training data was of "medium" size.

Discussion and Conclusion

Side-tuning addresses the bias-variance tradeoff inherent in transfer learning. It leverages the "useful bias" of a large pre-trained model while ensuring "asymptotic consistency"—the ability to eventually master a task given enough data—through the side network.

Its primary advantages include:

  • Simplicity: It avoids the complex regularization schemes required by substitutive methods like EWC.
  • Zero Forgetting: The frozen base model ensures the original task performance remains untouched.
  • Flexibility: The black-box nature allows for easy integration with any pre-trained architecture.

While the current implementation assumes task identity is known to select the appropriate side network, future work may explore automatic task detection and the simultaneous use of multiple base models to further enhance knowledge transfer. Side-tuning stands as a powerful, efficient baseline for the growing field of network adaptation.

Summary updated

Abstract

When training a neural network for a desired task, one may prefer to adapt a pre-trained network rather than starting from randomly initialized weights. Adaptation can be useful in cases when training data is scarce, when a single learner needs to perform multiple tasks, or when one wishes to encode priors in the network. The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. In this paper, we propose a straightforward alternative: side-tuning. Side-tuning adapts a pre-trained network by training a lightweight “side" network that is fused with the (unchanged) pre-trained network via summation. This simple method works as well as or better than existing solutions and it resolves some of the basic issues with fine-tuning, fixed features, and other common approaches. In particular, side-tuning is less prone to overfitting, is asymptotically consistent, and does not suffer from catastrophic forgetting in incremental learning. We demonstrate the performance of side-tuning under a diverse set of scenarios, including incremental learning (iCIFAR, iTaskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results.
BibTeX
@inproceedings{zhang2020side,
  author = {Jeffrey O. Zhang and Alexander Sax and Amir Zamir and Leonidas J. Guibas and Jitendra Malik},
  title = {{Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58580-8\_41},
}
Figure