Introduction
Ensemble learning—the process of combining predictions from multiple models—is a well-established technique for improving the generalization and robustness of deep learning systems. However, as modern architectures scale to billions of parameters, the computational cost of training multiple independent models from scratch has become a significant bottleneck. Existing low-cost alternatives, such as Dropout-based pseudo-ensembles or temporal ensembles, often suffer from high correlation between members, limiting their effectiveness.
This paper introduces Prune and Tune Ensembles (PAT), a high-performance, resource-efficient framework. PAT creates diverse ensembles by training a single "parent" model, cloning it, and applying unique pruning masks to generate sparse "children." By briefly fine-tuning these children, the method achieves state-of-the-art accuracy and uncertainty estimation at a fraction of the cost of traditional ensembles.
Methodology: The PAT Workflow
The PAT approach consists of three distinct phases designed to maximize diversity while minimizing training overhead:
- Parent Training: A standard "parent" network is trained to convergence on the target task. This provides a high-quality initialization for all ensemble members.
- Diverse Pruning: The parent is cloned into $M$ child networks. Each child is then pruned to a specific sparsity level. To ensure diversity, the authors employ Anti-Random Pruning, where masks are chosen to be maximally distant. For instance, in a 50% sparse ensemble of two, Child A might retain the first half of the parameters while Child B retains the second half, ensuring full coverage of the parent's knowledge without redundancy.
- One-Cycle Tuning: Each sparse child is fine-tuned for a short duration (typically 1–10 epochs). A One-Cycle learning rate schedule is used to "push" the child model out of the parent's local basin and into a new, unique local optimum defined by its specific connectivity.
Sparsity as a Catalyst for Diversity
A key insight of this work is that sparsity acts as a "diversity engine." By removing different sets of weights, each child inherits a unique topological structure. This ensures that even though they share a common ancestry, they navigate different optimization paths during the tuning phase.
Experimental Evaluation
The authors evaluated PAT on standard computer vision benchmarks, including CIFAR-10 and CIFAR-100, using architectures like ResNet-18 and WideResNet-28-10. To test robustness and uncertainty estimation, they utilized corrupted versions of these datasets (CIFAR-10/100-C).
Key Results
- Accuracy: On CIFAR-100, a PAT ensemble with 6 members achieved 82.67% accuracy, outperforming other efficient methods like Snapshot Ensembles (82.1%) and Fast Geometric Ensembling (82.3%).
- Efficiency: PAT achieved these gains using only 0.85x the FLOPs of a single dense model's full training run. In contrast, methods like MIMO or BatchEnsemble often require significantly more compute.
- Uncertainty and Robustness: PAT demonstrated superior Expected Calibration Error (ECE) on corrupted data, indicating better-calibrated confidence compared to single models or dropout-based approaches.
Conclusion and Future Work
Prune and Tune Ensembles provide a practical path toward deploying high-quality ensembles in resource-constrained environments. By leveraging the unique loss landscapes created by pruning, PAT achieves the diversity of independent models with the efficiency of a single-model budget. Future research may explore the application of anti-random pruning to large-scale datasets like ImageNet and other domains such as Natural Language Processing.
Summary updated