Abstract
Modular neural architectures are gaining attention for their powerful generalization and efficient adaptation to new domains. However, training these models poses challenges due to optimization difficulties arising from intrinsic sparse connectivity. Leveraging knowledge from monolithic models through techniques like knowledge distillation can facilitate training and enable integration of diverse knowledge. Nevertheless, conventional knowledge distillation approaches are not tailored to modular models and struggle with unique architectures and enormous parameter counts. Motivated by these challenges, we propose module-to-module knowledge distillation (m2mKD) for transferring knowledge between modules. m2mKD combines teacher modules of a pretrained monolithic model and student modules of a modular model with a shared meta model respectively to encourage the student module to mimic the behaviour of the teacher module. We evaluate m2mKD on two modular neural architectures: Neural Attentive Circuits (NACs) and Vision Mixture-of-Experts (V-MoE). Applying m2mKD to NACs yields significant improvements in IID accuracy on Tiny-ImageNet (up to 5.6%) and OOD robustness on Tiny-ImageNet-R (up to 4.2%). Additionally, the V-MoE-Base model trained with m2mKD achieves 3.5% higher accuracy than end-to-end training on ImageNet-1k.
Introduction
Modular neural architectures, such as Mixture-of-Experts (MoE) and Neural Attentive Circuits (NACs), represent a shift towards more flexible and efficient AI systems. By decomposing large-scale models into independent, specialized modules, these architectures can achieve superior generalization and computational efficiency. However, the sparse connectivity inherent in these models—where only a subset of parameters is active for any given input—presents significant optimization hurdles. Training these models from scratch often results in slow convergence or suboptimal performance.
Knowledge Distillation (KD) is a well-established technique for improving the training of smaller "student" models by having them mimic a larger "teacher" model. However, traditional KD methods are designed for monolithic architectures and struggle when applied to modular systems. The primary issues include a significant capacity gap between modular students and monolithic teachers, and an architectural mismatch that standard KD cannot bridge. To address these issues, we introduce m2mKD (Module-to-Module Knowledge Distillation).
Methodology: The m2mKD Framework
The m2mKD approach breaks down the distillation process into three distinct stages: Preparation, Module-to-Module Distillation, and End-to-End Training.
1. Preparation: Deep Incubation
The first stage involves preparing a monolithic teacher model so that its components are optimized to function as independent modules. This is achieved through a process called "Deep Incubation." Instead of training a full model, a small "meta-model" (a skeleton architecture) is used to host and train individual sub-modules. This ensures that the teacher's knowledge is already structured in a modular fashion, making it a more effective "instructor" for a modular student.
2. m2mKD: Module-to-Module Distillation
The core innovation of this work is the module-to-module distillation step. To train the student modules, "hybrid" networks are constructed. A specific layer in the meta-model is replaced with either a teacher module or a student module. The student module is then trained to minimize the difference between its output and the teacher's output within this shared context.
To handle potential dimension mismatches between the meta-model and the student modules, the framework employs temporary "stitch layers." These linear layers align the data representations during distillation and are discarded once the process is complete. This allows for parallel distillation of individual modules, which is highly efficient for large-scale models.
3. End-to-End (E2E) Training
In the final stage, the distilled student modules are integrated into the full modular architecture. The entire model is then fine-tuned on the target task. Because the modules have already been "pre-trained" to capture essential features from the teacher, the final end-to-end training converges much faster and reaches higher performance levels than training from scratch.
Experimental Results
The researchers evaluated m2mKD across several benchmarks, including ImageNet-1k, Tiny-ImageNet, and robustness tests like Tiny-ImageNet-R.
- Neural Attentive Circuits (NACs): On Tiny-ImageNet, m2mKD improved IID accuracy by up to 5.6%. Furthermore, it significantly enhanced out-of-distribution (OOD) robustness, with a 4.2% improvement on Tiny-ImageNet-R.
- Vision Mixture-of-Experts (V-MoE): The V-MoE-Base model trained with m2mKD achieved 81.9% accuracy on ImageNet-1k. This is a 3.5% improvement over standard end-to-end training and a massive jump over standard KD, which only reached 76.5%.
Crucially, m2mKD proved to be efficient. The total training time, including the preparation and distillation stages, was comparable to or even slightly faster than pure end-to-end training due to the accelerated convergence of the pre-distilled modules.
Conclusion
m2mKD offers a robust solution for the optimization challenges of modular transformers. By shifting the focus from whole-model distillation to module-level knowledge transfer, it enables the effective training of large-scale, sparse architectures. While the current pipeline involves multiple stages, its parallelizable nature and significant performance gains make it a promising framework for the next generation of modular AI. Future work will focus on simplifying the preparation phase and exploring the use of existing pretrained models as teachers.
Summary updated