Abstract
A standard assumption in deep learning is that the inductive bias introduced by a neural network architecture must persist from training through inference. The architecture you train with is the architecture you deploy. This assumption constrains the community from selecting architectures that may have desirable efficiency or design properties due to difficulties with optimization. We challenge this assumption with Network of Theseus (NoT), a method for progressively converting a trained, or even untrained, guide network architecture part-by-part into an entirely different target network architecture while preserving the performance of the guide network. At each stage, components in the guide network architecture are incrementally replaced with target architecture modules and aligned via representational similarity metrics. This procedure largely preserves the functionality of the guide network even under substantial architectural changes-for example, converting a convolutional network into a multilayer perceptron, or GPT-2 into a recurrent neural network. By decoupling optimization from deployment, NoT expands the space of viable inference-time architectures, opening opportunities for better accuracy-efficiency tradeoffs and enabling more directed exploration of the architectural design space.
Introduction
In modern deep learning, researchers are typically bound by a "what you train is what you get" paradigm. If the efficiency of a Recurrent Neural Network (RNN) or the specific inductive biases of a Convolutional Neural Network (CNN) are desired at test time, that specific architecture must generally be trained from scratch. However, many architectures optimized for deployment—such as deep MLPs or RNNs—are notoriously "optimization-unfriendly," making them difficult to train to high accuracy using standard supervised learning. Conversely, architectures that are easy to train, like large Transformers or ResNets, may be too computationally expensive for edge deployment.
The Network of Theseus (NoT) challenges the assumption that training and inference architectures must be identical. Inspired by the philosophical paradox of a ship whose planks are replaced one by one until none of the original components remain, NoT seeks to transfer the "soul" (the learned function and inductive biases) of a guide network into a more desirable target network.
Methodology
The NoT framework decouples optimization from deployment through a two-step process: Progressive Layer Replacement and Standard Training.
- Part-by-Part Replacement: Instead of replacing the entire network at once, NoT swaps out individual layers or blocks according to a Progressive Replacement Schedule. This prevents error accumulation that occurs when replacing components independently.
- Representational Alignment: At each stage, the new target module is optimized to match the internal activations (representations) of the corresponding module in the guide network.
- Similarity Metrics: To ensure high-fidelity mimicry, NoT utilizes Centered Kernel Alignment (CKA) and a new metric called Differentiable Mutual Nearest Neighbors (D-MNN), which focuses on preserving the local geometric structure of the data in the representation space.
- Final Fine-tuning: Once the entire "ship" has been replaced by the target architecture, the model undergoes a final round of standard end-to-end training on the downstream task to recover performance.
Experimental Results
The researchers evaluated NoT across dramatic architectural shifts in both Computer Vision (ImageNet) and Language Modeling (Wikitext-103).
As shown in Figure 5, NoT maintains significantly higher accuracy than "no alignment" baselines during the conversion of a ResNet-18 into a deep MLP. While MLPs are traditionally difficult to train on ImageNet, NoT allows them to represent complex functions by guiding them through the optimization process.
| Conversion | Guide Performance | NoT Performance | Baseline (From Scratch) |
|---|---|---|---|
| ResNet-18 → MLP (ImageNet) | 69.66% | 62.12% | 33.36% |
| DINOv2 → Patch-MLP (ImageNet) | 81.03% | 72.56% | 65.55% |
| GPT-2 → RNN (Wikitext-103) | 37.50 (Perp) | 50.58 (Perp) | 121.19 (Perp) |
Note: For Perplexity (Perp), lower is better.
Key Insights
- Inductive Bias is Transferable: Surprisingly, NoT works even when the guide network is untrained. An untrained ResNet-50 guide can help a ResNet-18 target reach higher accuracy than training from scratch, suggesting that the architectural structure itself provides a useful "scaffold" for optimization.
- The Progressive Advantage: Replacing layers sequentially while jointly tuning previously replaced layers is far superior to replacing them independently or all at once. This iterative alignment is crucial for maintaining the integrity of the learned function.
- Identifying Bottlenecks: By monitoring alignment loss, researchers can identify "bottleneck layers"—specific parts of the architecture (like downsampling layers) that are significantly harder to convert than others.
Limitations and Future Work
While NoT vastly outperforms baselines, a performance gap remains between the original guide and the final target. Additionally, the multi-stage replacement process increases computational costs compared to standard training. Future work will focus on scaling NoT to frontier models (e.g., 70B+ parameter LLMs) and refining alignment techniques to close the performance gap further.
Summary updated