A Flexible Selection Scheme for Minimum-Effort Transfer Learning

Amelie Royer, Christoph H. Lampert

Abstract

Fine-tuning is a popular way of exploiting knowledge contained in a pre-trained convolutional network for a new visual recognition task. However, the orthogonal setting of transferring knowledge from a pretrained network to a visually different yet semantically close source is rarely considered: This commonly happens with real-life data, which is not necessarily as clean as the training source (noise, geometric transformations, different modalities, etc.). To tackle such scenarios, we introduce a new, generalized form of fine-tuning, called flex-tuning, in which any individual unit (e.g. layer) of a network can be tuned, and the most promising one is chosen automatically. In order to make the method appealing for practical use, we propose two lightweight and faster selection procedures that prove to be good approximations in practice. We study these selection criteria empirically across a variety of domain shifts and data scarcity scenarios, and show that fine-tuning individual units, despite its simplicity, yields very good results as an adaptation technique. As it turns out, in contrast to common practice, rather than the last fully-connected unit it is best to tune an intermediate or early one in many domain-shift scenarios, which is accurately detected by flex-tuning.

Introduction

Transfer learning typically involves fine-tuning either the final classifier head or the entire network. While effective for new tasks with similar visual domains, these strategies often falter when the labels remain consistent but the visual domain shifts—such as moving from natural photos to sketches or dealing with sensor noise. In these "orthogonal" domain shift scenarios, full fine-tuning risks overfitting on small target datasets, while tuning only the last layer may be insufficient to bridge the domain gap.

Flex-tuning: Automated Layer Selection

The authors propose flex-tuning, a generalized fine-tuning framework that automatically identifies the single most effective unit (layer or block) to adapt for a given domain shift. By treating the choice of layer as a model selection problem, flex-tuning finds a "Goldilocks" solution that balances adaptation capacity with regularization.

To make this practical, three variants are introduced:

  1. Flex-tuning (Exhaustive): Trains and evaluates models for every possible single-unit tuning scenario.
  2. Fast Flex-tuning: Uses "network surgery" by fine-tuning the whole network once and evaluating "proxy" models where only one layer's weights are updated.
  3. Faster Flex-tuning: A further optimization where the initial full fine-tuning is limited to a single epoch.

Experimental Evaluation

The method was tested across a wide range of domain shifts, from simple synthetic noise to complex stylistic changes.

Examples of source and target domains used in the study, ranging from MNIST digits to ILSVRC natural images and their transformed counterparts.

The datasets included:

  • MNIST/CIFAR-10: Adapted to blurred, noisy, and sketched versions (QuickDraw).
  • ILSVRC (ImageNet): Adapted to color shifts (YUV/HSV), geometric transformations (rotation/scaling), and the PACS dataset (Art, Cartoon, Sketch).

Key Insights and Results

A central finding is that the standard practice of tuning the last layer is rarely optimal. The optimal layer to tune is highly dependent on the nature of the domain shift.

Visualization of the layers selected by flex-tuning variants across different tasks. Early layers are often chosen for pixel-level shifts, while deeper layers are selected for stylistic shifts.
  • Shift-Layer Correlation: Pixel-level shifts (e.g., color) are best addressed by early layers, geometric shifts by middle layers, and stylistic shifts by later layers.
  • Data Efficiency: Flex-tuning excels in low-data regimes (e.g., 2-20 images per class) because restricting updates to a single unit prevents the catastrophic overfitting common in full fine-tuning.
  • Performance: Across 72 scenarios, flex-tuning or its fast variants outperformed baselines in 60 cases, often by significant margins.

Conclusion

Flex-tuning offers a lightweight yet powerful alternative to standard transfer learning recipes. By automatically selecting the right depth for adaptation, it provides superior performance and robustness to data scarcity with minimal computational overhead when using the proposed fast selection procedures.

Summary updated

Abstract

Fine-tuning is a popular way of exploiting knowledge contained in a pre-trained convolutional network for a new visual recognition task. However, the orthogonal setting of transferring knowledge from a pretrained network to a visually different yet semantically close source is rarely considered: This commonly happens with real-life data, which is not necessarily as clean as the training source (noise, geometric transformations, different modalities, etc.). To tackle such scenarios, we introduce a new, generalized form of fine-tuning, called flex-tuning, in which any individual unit (e.g. layer) of a network can be tuned, and the most promising one is chosen automatically. In order to make the method appealing for practical use, we propose two lightweight and faster selection procedures that prove to be good approximations in practice. We study these selection criteria empirically across a variety of domain shifts and data scarcity scenarios, and show that fine-tuning individual units, despite its simplicity, yields very good results as an adaptation technique. As it turns out, in contrast to common practice, rather than the last fully-connected unit it is best to tune an intermediate or early one in many domain-shift scenarios, which is accurately detected by flex-tuning.
BibTeX
@inproceedings{royer2020flexible,
  author = {Amelie Royer and Christoph H. Lampert},
  title = {{A Flexible Selection Scheme for Minimum-Effort Transfer Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093635},
}
Figure