Abstract
Fine-tuning is a popular way of exploiting knowledge contained in a pre-trained convolutional network for a new visual recognition task. However, the orthogonal setting of transferring knowledge from a pretrained network to a visually different yet semantically close source is rarely considered: This commonly happens with real-life data, which is not necessarily as clean as the training source (noise, geometric transformations, different modalities, etc.). To tackle such scenarios, we introduce a new, generalized form of fine-tuning, called flex-tuning, in which any individual unit (e.g. layer) of a network can be tuned, and the most promising one is chosen automatically. In order to make the method appealing for practical use, we propose two lightweight and faster selection procedures that prove to be good approximations in practice. We study these selection criteria empirically across a variety of domain shifts and data scarcity scenarios, and show that fine-tuning individual units, despite its simplicity, yields very good results as an adaptation technique. As it turns out, in contrast to common practice, rather than the last fully-connected unit it is best to tune an intermediate or early one in many domain-shift scenarios, which is accurately detected by flex-tuning.
Introduction
Transfer learning typically involves fine-tuning either the final classifier head or the entire network. While effective for new tasks with similar visual domains, these strategies often falter when the labels remain consistent but the visual domain shifts—such as moving from natural photos to sketches or dealing with sensor noise. In these "orthogonal" domain shift scenarios, full fine-tuning risks overfitting on small target datasets, while tuning only the last layer may be insufficient to bridge the domain gap.
Flex-tuning: Automated Layer Selection
The authors propose flex-tuning, a generalized fine-tuning framework that automatically identifies the single most effective unit (layer or block) to adapt for a given domain shift. By treating the choice of layer as a model selection problem, flex-tuning finds a "Goldilocks" solution that balances adaptation capacity with regularization.
To make this practical, three variants are introduced:
- Flex-tuning (Exhaustive): Trains and evaluates models for every possible single-unit tuning scenario.
- Fast Flex-tuning: Uses "network surgery" by fine-tuning the whole network once and evaluating "proxy" models where only one layer's weights are updated.
- Faster Flex-tuning: A further optimization where the initial full fine-tuning is limited to a single epoch.
Experimental Evaluation
The method was tested across a wide range of domain shifts, from simple synthetic noise to complex stylistic changes.
The datasets included:
- MNIST/CIFAR-10: Adapted to blurred, noisy, and sketched versions (QuickDraw).
- ILSVRC (ImageNet): Adapted to color shifts (YUV/HSV), geometric transformations (rotation/scaling), and the PACS dataset (Art, Cartoon, Sketch).
Key Insights and Results
A central finding is that the standard practice of tuning the last layer is rarely optimal. The optimal layer to tune is highly dependent on the nature of the domain shift.
- Shift-Layer Correlation: Pixel-level shifts (e.g., color) are best addressed by early layers, geometric shifts by middle layers, and stylistic shifts by later layers.
- Data Efficiency: Flex-tuning excels in low-data regimes (e.g., 2-20 images per class) because restricting updates to a single unit prevents the catastrophic overfitting common in full fine-tuning.
- Performance: Across 72 scenarios, flex-tuning or its fast variants outperformed baselines in 60 cases, often by significant margins.
Conclusion
Flex-tuning offers a lightweight yet powerful alternative to standard transfer learning recipes. By automatically selecting the right depth for adaptation, it provides superior performance and robustness to data scarcity with minimal computational overhead when using the proposed fast selection procedures.
Summary updated