Superposing many tickets into one: A performance booster for sparse neural network training

Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy

Abstract

Recent works on sparse neural network training have shown that a compelling trade-off between performance and efficiency can be achieved. Existing sparse training methods usually strive to find the best possible subnetwork in a single training run. This paper introduces Sup-tickets, a novel approach that identifies multiple high-performing sparse configurations ("cheap tickets") during the final stages of training and superposes them into a single "ultimate ticket." This method significantly boosts performance across various architectures and datasets without increasing inference costs.

Introduction

Training sparse neural networks from scratch is a critical goal for efficient AI, aiming to bypass the memory and computational overhead of dense models. However, a performance gap remains: sparse models often fail to match the accuracy of dense counterparts, particularly at high sparsity levels (e.g., 95% of weights removed).

Traditional Dynamic Sparse Training (DST) methods focus on finding a single optimal subnetwork. The authors of "Sup-tickets" argue that this approach is limited. By leveraging the fact that neural network loss landscapes contain multiple low-loss basins, they propose a strategy to find several good sparse solutions within a single training trajectory and merge them into a more robust model.

Methodology: The Sup-tickets Framework

The Sup-tickets approach is a two-phase training strategy designed to be integrated into existing sparse training algorithms like RigL or SET.

  1. Exploration Phase (First 90%): The model undergoes standard sparse training to reach a stable, high-performing region of the loss landscape.
  2. Cyclical Phase (Last 10%): The training switches to a cyclical learning rate schedule. This forces the model to "jump" between nearby low-loss basins. At the end of each cycle, a snapshot of the sparse weights is taken, representing a "cheap ticket."
  3. Superposition and Pruning: These cheap tickets are averaged using Connection Independent Averaging (CIA). Since averaging multiple sparse masks can increase the number of active parameters, the resulting superposed model is pruned back to the target sparsity based on weight magnitude.
The Sup-tickets training pipeline: identifying cheap tickets through cyclical learning rates and superposing them into an ultimate ticket.

The final product is the "Ultimate Ticket": a single sparse subnetwork that maintains the inference efficiency of a standard sparse model while benefiting from the ensemble-like properties of the superposition.

Results and Analysis

The researchers evaluated Sup-tickets on standard benchmarks including CIFAR-10/100 and ImageNet using architectures like VGG-16 and ResNet-50.

Performance Gains

Sup-tickets demonstrated consistent improvements in 63 out of 66 tested scenarios. Notably, on CIFAR-100, the method combined with GraNet actually outperformed dense baselines while using only 10-20% of the parameters. On ImageNet, Sup-tickets improved the Top-1 accuracy of RigL by approximately 1.0% at high sparsity levels.

Performance comparison of different averaging methods (CIMA vs. CIA) across various sparsity levels for ResNet-50 and VGG-16 on CIFAR-100.

Key Insights

  • Averaging as a Booster: The results show that averaging sparse tickets found along a single trajectory acts as a powerful performance booster that can be applied to almost any sparse training algorithm with negligible overhead (less than 1% of total training FLOPs).
  • Diversity and Calibration: The diversity among "cheap tickets," even within the same general basin, allows the superposition to cancel out individual errors. This leads to better-calibrated models that are less prone to overconfidence.
  • Inference Efficiency: Unlike traditional ensemble methods that require running multiple models at inference time, Sup-tickets merges the ensemble into a single model, preserving the speed and memory benefits of sparsity.

Conclusion and Future Work

Sup-tickets offers a simple yet effective way to close the performance gap between sparse and dense neural networks. While it provides clear empirical benefits, questions remain regarding the internal representations learned by these superposed networks. Future research may explore how these models differ fundamentally from dense counterparts and how hardware can be better optimized to leverage unstructured sparsity for real-world speedups.

Summary updated

Abstract

Recent works on sparse neural network training have shown that a compelling trade-off between performance and efficiency can be achieved. Existing sparse training methods usually strive to find the...
BibTeX
@inproceedings{yin2022superposing,
  author = {Lu Yin and Vlado Menkovski and Meng Fang and Tianjin Huang and Yulong Pei and Mykola Pechenizkiy},
  title = {{Superposing many tickets into one: A performance booster for sparse neural network training}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=Y7N8qTlBN9},
}
Figure