TL;DR
This paper introduces "Sup-tickets," a novel method for sparse neural network training that significantly boosts performance by superposing multiple "cheap tickets" (low-loss subnetworks) found during training. This approach achieves state-of-the-art results, even outperforming dense networks at extreme sparsity levels, without increasing the overall training time.
What Problem Does This Paper Address?
Training extremely large neural networks has led to remarkable progress in various AI tasks. However, these models are computationally expensive and energy-intensive. Sparse neural network training offers a promising solution by reducing model size and computational cost while aiming to maintain high accuracy. Existing sparse training methods often focus on finding a single optimal sparse subnetwork. This paper argues that instead of solely optimizing for one subnetwork, it's more effective to leverage the existence of multiple low-loss subnetworks ("cheap tickets") within the loss landscape. The challenge is to efficiently discover these tickets and combine them into a single, high-performing sparse network without incurring additional training costs.
Methods
The core of Sup-tickets lies in two key desiderata:
- Efficiently Creating Cheap Tickets: During the last 10% of the training time, Sup-tickets employs a cyclical learning rate schedule and parameter exploration (pruning and growing) to discover multiple low-loss subnetworks. This cyclical process allows the model to explore different regions of the loss landscape, generating diverse "cheap tickets." The cycle length is significantly reduced compared to prior methods to fit within the standard training time.
- Superposing Tickets: To combine these cheap tickets into a single, stronger subnetwork, Sup-tickets uses a weight averaging strategy. Specifically, they found that "Connection Independent Averaging" (CIA) performs best. This method averages the weights of all collected cheap tickets. To maintain the target sparsity, a final pruning step is applied to remove weights with the smallest magnitudes after averaging. This process is designed to be memory-efficient, requiring only one extra copy of the averaged weights.
Datasets
The paper evaluates Sup-tickets on the following datasets:
- CIFAR-10
- CIFAR-100
- ImageNet
Key Insights and Ideas
- Leveraging Multiple Low-Loss Basins: The paper's central idea is that the loss landscape of neural networks contains multiple low-loss regions (basins), and by exploring these, we can find better sparse subnetworks than by focusing on just one.
- "Cheap Tickets" for Efficiency: Discovering multiple sparse subnetworks ("cheap tickets") within the final 10% of training time is crucial for maintaining efficiency.
- Averaging for Robustness: Superposing these cheap tickets through weight averaging leads to a more robust and higher-performing final sparse network.
- CIA Outperforms Other Averaging Methods: The "Connection Independent Averaging" (CIA) method proved most effective, suggesting that occasionally activated connections are less important than those consistently present across multiple tickets.
- Universal Booster: Sup-tickets is presented as a general-purpose performance enhancer that can be seamlessly integrated with various existing sparse training methods.
Key Results
- Consistent Performance Improvement: Sup-tickets consistently improved the performance of various sparse training methods (SET, RigL, GraNet, ERK, SNIP) across different architectures (VGG-16, ResNet-50, Wide ResNet28-10) and datasets (CIFAR-10/100, ImageNet).
- Outperforming Dense Networks: When combined with GraNet, Sup-tickets achieved performance comparable to or better than dense networks, even at extreme sparsity levels (90%-95%). For example, on CIFAR-100, GraNet+Sup-tickets outperformed dense VGG-16.
- No Training Time Overhead: The method achieves these gains without increasing the overall training time, as ticket generation is confined to the last 10% of training.
- Improved Uncertainty Estimation: Sup-tickets also demonstrated improved uncertainty estimation (ECE and NLL) compared to baseline sparse training methods.
- Effectiveness of Averaging: The paper showed that averaging subnetworks with high variance significantly degraded performance, highlighting the importance of finding tickets within similar low-loss basins.
Limitations and Future Work
- Diversity of Cheap Tickets: While Sup-tickets aims for diversity, the paper notes that the diversity of cheap tickets obtained by their method is lower than traditional dense ensembles. However, they argue this controlled diversity is beneficial for weight averaging.
- Representation Learning: The authors pose a question for future work: "even if Sup-tickets enable sparse neural networks to match or outperform their dense counterparts in terms of test accuracy, do they learn the same representation as the latter learn?"
- Hardware Acceleration: The authors hope their work inspires research into developing hardware accelerators better suited for sparse training.
- Extending Training Time: While the core method works within fixed training time, experiments showed that extending training time to generate more tickets further improves performance.
Summary updated