Early Identification of Sparse Winning Tickets

This direction is split into other directions: early-stage subnetwork identification. Its last version stays readable and citable here.

Version 1 of 186 papers135 conceptsfrozen 2026-10-04

The "Lottery Ticket Hypothesis" (LTH) famously suggests that dense neural networks contain sparse subnetworks—"winning tickets"—that can be trained in isolation to match the performance of the original model. However, the traditional method for finding these tickets is computationally expensive, often requiring a full cycle of training, pruning, and retraining.

A growing body of research suggests we don't have to wait for a model to converge to find these tickets. Instead, the essential structural connectivity of a network stabilizes very early in the training process, allowing us to identify "Early-Bird" (EB) tickets and drastically reduce the cost of deep learning.

The Intuition: Structure Converges Faster Than Weights

In the early phases of training, a neural network undergoes a "chaotic" period where it rapidly identifies the most important features of the data. Researchers like Haoran You and Xiaohan Chen have observed that the network's connectivity pattern—the binary mask of which weights are "important"—reaches a stable state long before the actual weight values converge.

Think of it like building a skyscraper: the steel skeleton (the structure) is bolted into place long before the interior finishes (the weights) are finalized. Once the skeleton is stable, you know exactly which parts of the building are load-bearing and which are decorative. In Drawing early-bird tickets, Haoran You demonstrates that these structural "skeletons" emerge within the first 10% to 20% of training.

Detecting Stability via Mask Distance

How do we know when a "winning ticket" has emerged? The key is to monitor the pruning mask—a binary matrix where 1 represents a kept weight and 0 represents a pruned one.

By calculating the Hamming distance or cosine similarity between masks at successive epochs, researchers can detect when the network's architecture has stopped shifting. When the distance between masks becomes sufficiently small, the structure is considered stable. Wenyu Sun and Xiangcheng Liu use this stability to trigger pruning in their OFARPruning framework, allowing them to identify high-performing sub-networks early and bypass the remainder of the dense training cycle.

Similarly, Maying Shen and Pavlo Molchanov proposed the Early Pruning Indicator (EPI), a metric that identifies the precise moment structural importance stabilizes, enabling a 2.4× reduction in training costs without sacrificing accuracy.

Beyond CNNs: A Universal Phenomenon

This early structural emergence isn't limited to simple image classifiers. It appears to be a fundamental property of how various architectures learn:

  • Transformers: In EarlyBERT, Xiaohan Chen and Yu Cheng 0001 show that redundant attention heads and neurons can be pruned within the first few percent of training, slashing BERT training time by up to 45%.
  • Graph Neural Networks (GNNs): Haoran You and Zhihan Lu found that both the graph topology and the model weights in GCNs converge to a stable sparse state within the first quarter of training.
  • Diffusion Models: Even massive generative models exhibit this behavior. L. R. Whalen and Haoran You demonstrated in Early-Bird Diffusion that connectivity patterns stabilize within the first 0.5% to 2.5% of the total training duration.

Efficiency and Robustness

The primary benefit of finding tickets early is efficiency. By pruning the model early, we can perform the majority of the training on a much smaller, faster architecture. However, there is also evidence that early pruning acts as a form of regularization.

Zhiheng Xi and Rui Zheng leveraged this in EarlyRobust, identifying "robust early-bird tickets" to speed up adversarial training by 13x. They argue that the essential connectivity required for resilience is captured early, while later stages of dense training might actually lead the model to overfit to noise.

The Future of Zero-Shot Discovery

While "Early-Bird" methods significantly reduce costs, the ultimate goal is zero-shot pruning—finding the winning ticket at initialization without any training at all.

While this remains a challenge, work by Yanwei Fu on DessiLBI and Milad Alizadeh on Prospect Pruning suggests that by simulating short-term learning dynamics or using meta-gradients, we can move closer to identifying these optimal structures before the first epoch even begins.

Papers in this version

Showing 60 of 86 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_early_identification_of_sparse_winning_t_d635170f,
  title = {Early Identification of Sparse Winning Tickets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/early-identification-of-sparse-winning-tickets/rdr_d635170fd62b4c6d9493fcea897735a1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_early_identification_of_sparse_winning_t_d635170f_v1,
  title = {Early Identification of Sparse Winning Tickets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/early-identification-of-sparse-winning-tickets/rdr_d635170fd62b4c6d9493fcea897735a1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}