Abstract
The Lottery Ticket Hypothesis (LTH) posits that dense neural networks contain sparse sub-networks that can be trained in isolation to match the performance of the original model. Traditionally, finding these "winning tickets" is a task-specific, iterative process. This paper investigates the uniqueness and transferability of these sub-networks. We demonstrate that winning tickets are not unique and that different pruning methods identify distinct but equally performant sub-networks. By intersecting masks discovered across multiple tasks, we introduce "Prêt-à-Porter" tickets—robust, highly sparse sub-networks that can be found in parallel and generalize effectively across diverse datasets.
Introduction
The discovery of "winning tickets" within over-parameterized neural networks has sparked significant interest in network pruning and initialization. However, the standard approach to finding these tickets is "bespoke": it requires iterative training and pruning for a specific architecture and task. This process is computationally intensive and raises fundamental questions about the nature of these sub-networks. Are they unique to a specific task-initialization pair, or do they represent a more general "core" structure?
This study explores the intersection of task-specific masks to identify shared structural components. We contrast "bespoke" tickets, tailored to individual tasks, with "Prêt-à-Porter" tickets, which are derived from a consensus of masks across multiple related tasks.
Methodology
The researchers evaluated 28 combinations of architectures and tasks. Bespoke tickets were generated using seven distinct pruning techniques, ranging from magnitude-based unstructured pruning to structured channel removal. Mask similarity was quantified using the Jaccard distance.
To find more generalizable sub-networks, the authors proposed a consensus-based method. Instead of sequential pruning on a single task, the same initial model is trained on $N$ different tasks in parallel. The resulting task-specific masks are then intersected. Only parameters identified as important across all tasks are retained in the final Prêt-à-Porter ticket.
Experimental Setup
The empirical study utilized four architectures: LeNet, AlexNet, VGG11, and ResNet18. These were tested across seven image classification datasets, including grayscale digits (MNIST, Fashion-MNIST, KMNIST, EMNIST), natural images (CIFAR-10, CIFAR-100), and house numbers (SVHN).
Results and Discussion
The investigation yielded several key insights:
- Non-Uniqueness of Winning Tickets: Different pruning methods (e.g., L1 vs. L2 magnitude pruning) produced masks with high Jaccard distance (low overlap) yet nearly identical test accuracy. This suggests that multiple disjoint winning tickets exist within the same initialization.
- Task-Driven Divergence: Masks sourced from different tasks using the same pruning method showed significant dissimilarity, especially at high sparsity levels. As models become sparser, the overlap between tasks tends to shrink toward zero.
- Performance of Prêt-à-Porter Tickets: Despite their extreme sparsity, Prêt-à-Porter tickets achieved performance comparable to bespoke tickets. For example, intersecting five bespoke masks for LeNet resulted in a ticket with ~85% sparsity that remained highly trainable.
- Initialization Stability: Weights in Prêt-à-Porter tickets tend to originate from a distribution with smaller variance at initialization compared to bespoke tickets, suggesting a "denoising" effect that contributes to training stability.
- Efficiency through Parallelism: Unlike the sequential nature of traditional LTH discovery, Prêt-à-Porter tickets can be found in parallel across multiple tasks, significantly reducing the time required for sub-network discovery.
Conclusion
This work challenges the notion of unique winning tickets and demonstrates that a "shared core" of parameters exists across related tasks. Prêt-à-Porter tickets offer a computationally efficient alternative to bespoke sub-networks, providing robust performance with higher sparsity. Future research should explore the limits of this consensus approach across more diverse domains and larger-scale models.
Summary updated