Towards Reusable Network Components by Learning Compatible Representations

Michael Gygli, Jasper R. R. Uijlings, Vittorio Ferrari

Abstract

This paper proposes a novel approach towards compatible and reusable neural network components. Rather than training networks for different tasks independently, we adapt the training process to produce components that are compatible across tasks. By splitting a network into a feature extractor and a target task head, we propose various approaches to achieve compatibility, including shared auxiliary tasks and identical weight initializations. We demonstrate that components can be directly swapped without fine-tuning or compromising accuracy. Applications include unsupervised domain adaptation, cross-architecture transfer, and efficient transfer learning.

1. Introduction

Modern computer vision typically treats neural networks as monolithic, task-specific blocks. This lack of modularity necessitates expensive retraining or fine-tuning when adapting models to new domains or architectures. We envision a modular ecosystem where building a model involves selecting functionally well-defined parts from a library. The primary obstacle is compatibility: ensuring a feature extractor from one model produces representations interpretable by a task head from another.

2. Methodology

We define a network as a composition of a feature extractor $f$ and a task head $h$. To enforce compatibility, we propose three strategies:

  1. Compatibility Through Self-Supervision (RP): Models share an auxiliary task, such as predicting image rotation. Sharing a rotation-prediction head forces feature extractors to align their internal representations.
  2. Compatibility Through Discriminating Common Classes (DCC): For datasets with overlapping categories, a shared auxiliary head classifies these common classes across both networks.
  3. Compatibility Through Identical Initial Weights (IIW): Starting training from identical weights anchors optimization in the same region of the feature space, providing a baseline level of compatibility.
Joint training scheme using a shared auxiliary head to enforce compatibility between different feature extractors.

These methods can be implemented via joint training (simultaneous) or incremental training (adapting a new component to a frozen one).

3. Experimental Analysis

We evaluated these methods using CIFAR-10 and STL-10, which share nine classes. Our results indicate that compatibility is distinct from feature similarity. While self-supervision (RP) increases feature correlation, DCC often yields higher recombination accuracy.

Testing recombination by swapping feature extractors and task heads across models trained on STL-10 and CIFAR-10.

Combining IIW, RP, and DCC achieved a recombination accuracy of 76.7%, significantly outperforming independent training (10.6%) and approaching the upper bound of a task-specific model (85.6%).

4. Applications

Unsupervised Domain Adaptation (UDA)

Our framework matches state-of-the-art performance on CIFAR-10 $\leftrightarrow$ STL-10 adaptation while being significantly more efficient, requiring only 1,000 training steps compared to 80,000 for traditional methods.

Cross-Architecture Transfer

We successfully transferred a classification head from a Wide ResNet-56 to a MobileNet V2. This allows for deploying a single high-performance head across diverse hardware backbones without retraining the head for each specific architecture.

Efficient Transfer Learning

Compatible components enable "zero-shot" model selection. By snapping a source feature extractor onto a target head, performance can be estimated immediately, bypassing the need for expensive fine-tuning during the evaluation of multiple source models.

5. Conclusion

We have demonstrated that neural network components can be made compatible and reusable through targeted training strategies. This modular approach reduces the computational cost of adaptation and enables flexible cross-architecture deployment. Future work will explore universal auxiliary tasks and more robust handling of Batch Normalization statistics during component swapping.

Summary updated

Abstract

This paper proposes to make a first step towards compatible and hence reusable network components. Rather than training networks for different tasks independently, we adapt the training process to produce network components that are compatible across tasks. In particular, we split a network into two components, a features extractor and a target task head, and propose various approaches to accomplish compatibility between them. We systematically analyse these approaches on the task of image classification on standard datasets. We demonstrate that we can produce components which are directly compatible without any fine-tuning or compromising accuracy on the original tasks. Afterwards, we demonstrate the use of compatible components on three applications: Unsupervised domain adaptation, transferring classifiers across feature extractors with different architectures, and increasing the computational efficiency of transfer learning.
BibTeX
@inproceedings{gygli2021towards,
  author = {Michael Gygli and Jasper R. R. Uijlings and Vittorio Ferrari},
  title = {{Towards Reusable Network Components by Learning Compatible Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=kaOXOMJ10O},
}
Figure