TL;DR
Training large ensembles of deep neural networks is computationally expensive, limiting their practical use. MotherNets introduce a novel approach to train these ensembles much faster by sharing structural similarities between networks, creating a new "Pareto frontier" for the accuracy-training cost tradeoff. This allows for larger, more diverse ensembles with improved accuracy and significantly reduced training time.
What Problem Does This Paper Address?
Deep neural network ensembles are known to significantly boost generalization accuracy, making them valuable for tasks like image segmentation and drug discovery. However, the primary bottleneck is the prohibitive cost and time required to train each individual network in the ensemble. Current methods either train all networks from scratch, leading to massive computational demands and small ensemble sizes, or use monolithic architectures that sacrifice diversity and accuracy. This leaves a gap for methods that can efficiently train large, diverse ensembles without compromising performance.
Methods
The core innovation is the "MotherNet" concept. Instead of training each ensemble member independently, MotherNets first train one or more "MotherNets" that capture the structural commonalities within clusters of ensemble networks. These MotherNets are then used to "hatch" the individual ensemble networks, essentially transferring the learned function. The hatched networks then undergo a much faster fine-tuning process.
Key aspects of the method include:
- Clustering: Ensembles are clustered based on network architecture and topology. A separate MotherNet is trained for each cluster. This allows MotherNets to handle diverse architectures and provides a knob (number of clusters, g) to navigate the accuracy-training time tradeoff.
- MotherNet Construction: Algorithms are provided for constructing MotherNets for both fully-connected and convolutional networks. The MotherNet is designed to be the "shallowest" network that can encompass the structure of its cluster members.
- Hatching: Function-preserving transformations (specifically Net2Net) are used to expand the MotherNet into the target ensemble networks. This is a computationally inexpensive step.
- Fast Fine-tuning: The hatched networks, initialized from the MotherNet, converge significantly faster (within tens of epochs) compared to training from scratch.
Datasets
The paper evaluates MotherNets on several standard image classification datasets:
- CIFAR-10
- CIFAR-100
- SVHN
Key Insights and Ideas
- Shared "Epochs": The core idea is that MotherNets allow ensemble networks to implicitly "share epochs" by leveraging common structures, drastically reducing the total training time.
- New Pareto Frontier: MotherNets establish a new, superior Pareto frontier for the accuracy-training time tradeoff, outperforming existing fast ensemble training methods.
- Navigating the Tradeoff: The number of clusters (g) acts as a tunable parameter, allowing users to balance accuracy and training time based on their specific needs.
- Scalability: MotherNets demonstrate excellent scalability to very large ensembles (hundreds of models), maintaining performance gains where other methods falter.
- Diversity vs. Efficiency: While some diversity is sacrificed compared to fully independent training, MotherNets strike a better balance between diversity and computational efficiency than other fast ensemble methods.
Key Results
- Accuracy-Training Time Tradeoff: MotherNets achieve up to 2% better absolute test error rates compared to existing fast ensemble methods, often at comparable or lower training costs.
- Speedup: MotherNets can train ensembles significantly faster, with speedups of 2x to 4.2x compared to Knowledge Distillation and up to 3x reduction in cloud training costs.
- Scaling: For large ensembles (100 networks), MotherNets train significantly faster than Snapshot Ensembles, and their accuracy continues to improve while SE's degrades.
- Oracle Accuracy: MotherNets (with g=1) achieve better oracle test accuracy than Snapshot Ensembles.
- Model Covariance: MotherNets help reduce model covariance, contributing to better ensemble variance reduction compared to Snapshot Ensembles.
Limitations and Future Work
- Diversity Reduction: The paper acknowledges that MotherNets introduce some reduction in diversity and accuracy compared to training all networks independently.
- Architecture Class Limitations: For ensembles with highly diverse architecture classes, multiple MotherNets might be needed, increasing complexity.
- Future Work: The authors suggest that function-preserving transformations are an active area of research, and better techniques could be incorporated into MotherNets. They also mention exploring shared-MotherNets for further inference time and memory reduction, and parallel training strategies.
Summary updated