Training Neural Networks to Produce Compatible Features

Michael Gygli, Jasper R. R. Uijlings, Vittorio Ferrari

Abstract

This paper proposes to make a first step towards compatible and hence reusable network components. Rather than training networks for different tasks independently, we adapt the training process to produce network components that are compatible across tasks. In particular, we split a network into two components, a features extractor and a target task head, and propose various approaches to accomplish compatibility between them. We systematically analyse these approaches on the task of image classification on standard datasets. We demonstrate that we can produce components which are directly compatible without any fine-tuning or compromising accuracy on the original tasks. Afterwards, we demonstrate the use of compatible components on three applications: Unsupervised domain adaptation, transferring classifiers across feature extractors with different architectures, and increasing the computational efficiency of transfer learning.

Introduction

In contemporary computer vision, deep neural networks are predominantly trained as monolithic, task-specific entities. When a model needs to be adapted—for instance, transitioning from daylight to nighttime image recognition or swapping a heavy backbone for a mobile-friendly architecture—practitioners typically rely on extensive fine-tuning or complex "stitching" procedures. This paradigm is inherently inefficient.

The authors of this study envision a modular future for neural network design, akin to LEGO blocks or standardized automotive parts. In this vision, a high-quality feature extractor could be selected from a library and immediately paired with a specialized task head. The primary obstacle to this modularity is that independently trained networks develop unique internal representations, or "languages," making their components fundamentally incompatible by default. This paper introduces a framework to force networks to learn compatible internal representations during the initial training phase.

Methodology

To achieve "plug-and-play" compatibility between feature extractors and task heads, the researchers propose three primary strategies designed to align the internal representations of different models:

  1. Compatibility through Self-Supervision (RP): Models are trained on a shared auxiliary task, such as predicting the rotation of an image (0°, 90°, 180°, 270°). By sharing a common "rotation head," the feature extractors are incentivized to organize information in a consistent manner.
  2. Discriminating Common Classes (DCC): When tasks share categories (e.g., "dogs" or "cars" appearing in two different datasets), a shared auxiliary head is used to classify these commonalities, anchoring the feature space.
  3. Identical Initial Weights (IIW): Networks begin training from identical weight values, whether random or pre-trained. This ensures the optimization process starts from the same region in the loss landscape, significantly aiding convergence toward compatible representations.

The framework supports both Joint Training (optimizing multiple models simultaneously) and Incremental Training (aligning a new model with an existing, frozen component).

Applications and Empirical Evaluation

The researchers evaluated these methods using standard benchmarks including CIFAR-10, STL-10, CIFAR-100, and ImageNet (ILSVRC-12). The core finding is that compatibility does not require identical feature vectors; components merely need to be "understandable" by the same head.

Overview of applications for compatible components: (a) Analysis of component swapping, (b) Unsupervised Domain Adaptation, (c) Classifier Transferability across different feature extractors, and (d) Faster transfer learning on target datasets.

Key Results

  • Zero-Shot Recombination: While independently trained models fail when components are swapped (achieving near-chance accuracy), the combination of IIW, RP, and DCC allowed for 76.7% accuracy on CIFAR/STL swaps without any fine-tuning.
  • Unsupervised Domain Adaptation: The proposed method matched state-of-the-art performance while requiring significantly fewer computational steps—1,000 steps compared to the 80,000 steps required by traditional methods like VADA.
  • Cross-Architecture Transfer: The framework successfully enabled compatibility between different architectures, such as ResNet-56 and MobileNetV2, by utilizing a minimal 1x1 convolution "stitching" layer to align channel dimensions.
  • Transfer Learning Efficiency: When applied to new tasks like CIFAR-100, compatible components provided a "warm start," achieving instant accuracies of 40-66%, whereas standard transfer learning starts from near zero.

Conclusion

This research demonstrates that neural network components can be trained for compatibility without sacrificing performance on their original tasks. The study highlights that identical weight initialization (IIW) is a particularly powerful driver of compatibility, suggesting that early structural foundations in training are persistent.

Despite these successes, a "task mismatch" gap remains; a head optimized for one distribution (e.g., CIFAR-10) will not be perfectly optimal for another (e.g., STL-10) due to inherent data distribution shifts. Future work aims to establish standardized interfaces for neural components, moving the field closer to a truly modular ecosystem of interchangeable machine learning parts.

Summary updated

Abstract

This paper proposes to make a first step towards compatible and hence reusable network components. Rather than training networks for different tasks independently, we adapt the training process to produce network components that are compatible across tasks. In particular, we split a network into two components, a features extractor and a target task head, and propose various approaches to accomplish compatibility between them. We systematically analyse these approaches on the task of image classification on standard datasets. We demonstrate that we can produce components which are directly compatible without any fine-tuning or compromising accuracy on the original tasks. Afterwards, we demonstrate the use of compatible components on three applications: Unsupervised domain adaptation, transferring classifiers across feature extractors with different architectures, and increasing the computational efficiency of transfer learning.
BibTeX
@inproceedings{gygli2021training,
  author = {Michael Gygli and Jasper R. R. Uijlings and Vittorio Ferrari},
  title = {{Training Neural Networks to Produce Compatible Features}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=kaOXOMJ10O},
}
Figure