Abstract
This paper proposes a novel approach towards compatible and reusable neural network components. Rather than training networks for different tasks independently, we adapt the training process to produce components that are compatible across tasks. By splitting a network into a feature extractor and a target task head, we propose various approaches to achieve compatibility, including shared auxiliary tasks and identical weight initializations. We demonstrate that components can be directly swapped without fine-tuning or compromising accuracy. Applications include unsupervised domain adaptation, cross-architecture transfer, and efficient transfer learning.
1. Introduction
Modern computer vision typically treats neural networks as monolithic, task-specific blocks. This lack of modularity necessitates expensive retraining or fine-tuning when adapting models to new domains or architectures. We envision a modular ecosystem where building a model involves selecting functionally well-defined parts from a library. The primary obstacle is compatibility: ensuring a feature extractor from one model produces representations interpretable by a task head from another.
2. Methodology
We define a network as a composition of a feature extractor $f$ and a task head $h$. To enforce compatibility, we propose three strategies:
- Compatibility Through Self-Supervision (RP): Models share an auxiliary task, such as predicting image rotation. Sharing a rotation-prediction head forces feature extractors to align their internal representations.
- Compatibility Through Discriminating Common Classes (DCC): For datasets with overlapping categories, a shared auxiliary head classifies these common classes across both networks.
- Compatibility Through Identical Initial Weights (IIW): Starting training from identical weights anchors optimization in the same region of the feature space, providing a baseline level of compatibility.
These methods can be implemented via joint training (simultaneous) or incremental training (adapting a new component to a frozen one).
3. Experimental Analysis
We evaluated these methods using CIFAR-10 and STL-10, which share nine classes. Our results indicate that compatibility is distinct from feature similarity. While self-supervision (RP) increases feature correlation, DCC often yields higher recombination accuracy.
Combining IIW, RP, and DCC achieved a recombination accuracy of 76.7%, significantly outperforming independent training (10.6%) and approaching the upper bound of a task-specific model (85.6%).
4. Applications
Unsupervised Domain Adaptation (UDA)
Our framework matches state-of-the-art performance on CIFAR-10 $\leftrightarrow$ STL-10 adaptation while being significantly more efficient, requiring only 1,000 training steps compared to 80,000 for traditional methods.
Cross-Architecture Transfer
We successfully transferred a classification head from a Wide ResNet-56 to a MobileNet V2. This allows for deploying a single high-performance head across diverse hardware backbones without retraining the head for each specific architecture.
Efficient Transfer Learning
Compatible components enable "zero-shot" model selection. By snapping a source feature extractor onto a target head, performance can be estimated immediately, bypassing the need for expensive fine-tuning during the evaluation of multiple source models.
5. Conclusion
We have demonstrated that neural network components can be made compatible and reusable through targeted training strategies. This modular approach reduces the computational cost of adaptation and enables flexible cross-architecture deployment. Future work will explore universal auxiliary tasks and more robust handling of Batch Normalization statistics during component swapping.
Summary updated