Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding

Michela Paganini, Jessica Zosa FordearXiv 2020 · arXiv

Abstract

The Lottery Ticket Hypothesis (LTH) posits that dense neural networks contain sparse sub-networks that can be trained in isolation to match the performance of the original model. Traditionally, finding these "winning tickets" is a task-specific, iterative process. This paper investigates the uniqueness and transferability of these sub-networks. We demonstrate that winning tickets are not unique and that different pruning methods identify distinct but equally performant sub-networks. By intersecting masks discovered across multiple tasks, we introduce "Prêt-à-Porter" tickets—robust, highly sparse sub-networks that can be found in parallel and generalize effectively across diverse datasets.

Introduction

The discovery of "winning tickets" within over-parameterized neural networks has sparked significant interest in network pruning and initialization. However, the standard approach to finding these tickets is "bespoke": it requires iterative training and pruning for a specific architecture and task. This process is computationally intensive and raises fundamental questions about the nature of these sub-networks. Are they unique to a specific task-initialization pair, or do they represent a more general "core" structure?

This study explores the intersection of task-specific masks to identify shared structural components. We contrast "bespoke" tickets, tailored to individual tasks, with "Prêt-à-Porter" tickets, which are derived from a consensus of masks across multiple related tasks.

Methodology

The researchers evaluated 28 combinations of architectures and tasks. Bespoke tickets were generated using seven distinct pruning techniques, ranging from magnitude-based unstructured pruning to structured channel removal. Mask similarity was quantified using the Jaccard distance.

To find more generalizable sub-networks, the authors proposed a consensus-based method. Instead of sequential pruning on a single task, the same initial model is trained on $N$ different tasks in parallel. The resulting task-specific masks are then intersected. Only parameters identified as important across all tasks are retained in the final Prêt-à-Porter ticket.

Conceptual overview of Prêt-à-Porter mask discovery through the intersection of task-specific bespoke masks.

Experimental Setup

The empirical study utilized four architectures: LeNet, AlexNet, VGG11, and ResNet18. These were tested across seven image classification datasets, including grayscale digits (MNIST, Fashion-MNIST, KMNIST, EMNIST), natural images (CIFAR-10, CIFAR-100), and house numbers (SVHN).

Results and Discussion

The investigation yielded several key insights:

  1. Non-Uniqueness of Winning Tickets: Different pruning methods (e.g., L1 vs. L2 magnitude pruning) produced masks with high Jaccard distance (low overlap) yet nearly identical test accuracy. This suggests that multiple disjoint winning tickets exist within the same initialization.
  2. Task-Driven Divergence: Masks sourced from different tasks using the same pruning method showed significant dissimilarity, especially at high sparsity levels. As models become sparser, the overlap between tasks tends to shrink toward zero.
  3. Performance of Prêt-à-Porter Tickets: Despite their extreme sparsity, Prêt-à-Porter tickets achieved performance comparable to bespoke tickets. For example, intersecting five bespoke masks for LeNet resulted in a ticket with ~85% sparsity that remained highly trainable.
  4. Initialization Stability: Weights in Prêt-à-Porter tickets tend to originate from a distribution with smaller variance at initialization compared to bespoke tickets, suggesting a "denoising" effect that contributes to training stability.
  5. Efficiency through Parallelism: Unlike the sequential nature of traditional LTH discovery, Prêt-à-Porter tickets can be found in parallel across multiple tasks, significantly reducing the time required for sub-network discovery.
Test accuracy across different architectures and datasets as a function of pruned weight fraction.

Conclusion

This work challenges the notion of unique winning tickets and demonstrates that a "shared core" of parameters exists across related tasks. Prêt-à-Porter tickets offer a computationally efficient alternative to bespoke sub-networks, providing robust performance with higher sparsity. Future research should explore the limits of this consensus approach across more diverse domains and larger-scale models.

Summary updated

Abstract

The observation of sparse trainable sub-networks within over-parametrized networks - also known as Lottery Tickets (LTs) - has prompted inquiries around their trainability, scaling, uniqueness, and generalization properties. Across 28 combinations of image classification tasks and architectures, we discover differences in the connectivity structure of LTs found through different iterative pruning techniques, thus disproving their uniqueness and connecting emergent mask structure to the choice of pruning. In addition, we propose a consensus-based method for generating refined lottery tickets. This lottery ticket denoising procedure, based on the principle that parameters that always go unpruned across different tasks more reliably identify important sub-networks, is capable of selecting a meaningful portion of the architecture in an embarrassingly parallel way, while quickly discarding extra parameters without the need for further pruning iterations. We successfully train these sub-networks to performance comparable to that of ordinary lottery tickets.
BibTeX
@misc{paganini2020bespoke,
  author = {Michela Paganini and Jessica Zosa Forde},
  title = {{Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding}},
  year = {2020},
  eprint = {2007.04091},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.04091},
}
Figure