The Lottery Ticket Hypothesis, originally proposed by Jonathan Frankle in The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, demonstrated that dense neural networks contain sparse subnetworks ("winning tickets") capable of matching full-model performance when trained in isolation. However, finding these tickets is computationally expensive, requiring iterative training and pruning cycles. This limitation sparked a crucial question: Do these sparse architectures overfit to their specific training tasks, or do they possess universal properties that allow them to transfer across datasets and domains?
A growing body of research shows that winning tickets are not merely "lucky" for a single task. Instead, certain pruning regimes isolate structural inductive biases—topological masks—that capture fundamental properties of the underlying data domain, allowing a single sparse backbone to be reused across diverse downstream applications.
The Phenomenon of Universal Tickets
Early empirical evidence revealed that sparse subnetworks derived from large, diverse datasets contain generic inductive biases. In One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers, Ari S. Morcos et al. demonstrated that winning tickets generated from complex datasets like ImageNet can effectively initialize models for entirely different target tasks within the same domain, bypassing the need to repeat the expensive pruning process.
This cross-task structural transfer extends surprisingly far across disparate data distributions. For instance, Matthia Sabatelli et al. showed in On the Transferability of Winning Tickets in Non-Natural Image Datasets that winning tickets found on natural images (such as CIFAR-10) transfer successfully to highly specialized, non-natural domains like digital pathology and art history. In these data-scarce target environments, the transferred sparse models consistently outperform their larger, unpruned counterparts.
Similarly, in natural language processing, Tianlong Chen et al. discovered in The Lottery Ticket Hypothesis for Pre-trained BERT Networks that pre-trained language models contain universal subnetworks that transfer robustly to a wide variety of downstream NLP tasks without sacrificing accuracy.
Why Self-Supervision Isolates Transferable Structure
The transferability of a sparse mask is heavily influenced by the objective function used during the pruning phase. When a network is pruned using a narrow, task-specific supervised objective, the resulting subnetwork becomes highly specialized, locking in features that fail to generalize. Conversely, pruning via general, self-supervised proxy tasks isolates foundational representations.
In Pruning Convolutional Neural Networks with Self-Supervision, Mathilde Caron et al. demonstrated that sparse connectivity patterns discovered entirely without labels are highly competitive with those found via supervised learning, preserving feature quality for downstream transfer.
This principle is further validated in language and multimodal models. In Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Training, Yuanxin Liu et al. showed that training binary masks on task-agnostic pre-training objectives (like masked language modeling) yields highly transferable subnetworks. By forcing the mask to preserve the model's ability to perform its foundational, general-purpose tasks, the subnetwork retains the universal features necessary for diverse downstream applications. This mechanism explains why Zhe Gan et al. found universal subnetworks within large-scale vision-language models in Playing Lottery Tickets with Vision and Language, proving that multimodal interactions concentrate within specific, transferable architectural paths.
Loss Landscape Alignment and Flatness
Intuition about why these masks transfer can also be drawn from the geometry of the loss landscape. Optimization algorithms that actively seek out "flat minima"—regions where parameter perturbations do not drastically alter the loss—produce models that are inherently more resilient to compression.
As Clara Na et al. demonstrated in Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Models, identifying sparse subnetworks within flatness-optimized models yields structural masks that exhibit significantly higher transferability across downstream tasks than masks derived from standard point-wise optimization. Flatness-seeking optimization captures structural features that are more generally applicable across diverse domains.
Standard Optimization (Sharp Minimum) Flatness-Seeking Optimization (Flat Minimum)
| | | |
v v v v
Pruning causes severe Pruning causes minimal
performance degradation performance degradation
Furthermore, the loss landscapes of large pre-trained foundation models appear to be highly aligned across different applications. In A Second-Order Perspective on Pruning at Initialization and Knowledge Transfer, Leonardo Iurada et al. leveraged this alignment to show that a compression mask determined using data from a single source task preserves generalizable features and maintains high performance on unseen target tasks. Because the underlying loss landscapes share a common geometry, a mask that preserves the parameter flow for one task inherently preserves it for another.
The Physics of Transfer: A Renormalization Group Perspective
To establish a formal mathematical foundation for this universality, researchers have mapped the pruning process to concepts in theoretical physics. In Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective and Universality of Winning Tickets: A Renormalization Group Perspective, William T. Redman et al. proved that iterative magnitude pruning operates as a Renormalization Group (RG) flow in parameter space.
In physics, the RG framework explains how system behaviors change across different scales, grouping seemingly unrelated systems into "universality classes" based on shared scaling exponents. When applied to deep learning, this perspective reveals that tasks supporting successful subnetwork transfer share nearly identical parameter trajectories and scaling exponent distributions during the pruning process. When two tasks belong to the same universality class, their relevant parameter flow directions align. Consequently, a sparse architecture optimized for the parameter flow of a source task will inherently remain effective when repurposed for a target task within that same class.
Masks as Pure Inductive Biases
Ultimately, a transferable sparse subnetwork acts as a pure structural prior, encoding geometric and functional constraints directly into the network topology. This structural bias can sometimes be completely decoupled from specific weight values.
For example, Di He identified a "dominant winning ticket" in pre-trained language models—comprising as little as 0.05% of the total parameters—that acts as a task-agnostic skeleton for downstream adaptation in Finding the Dominant Winning Ticket in Pre-Trained Language Models.
This phenomenon manifests across multiple domains:
- Image Generation: In Optimal Eye Surgeon: Finding image priors through sparse generators at initialization, Avrajit Ghosh demonstrated that sparse masks optimized for image generation capture general natural image statistics rather than instance-specific details, allowing them to transfer seamlessly across different image categories.
- Graph Neural Networks: In Rethinking Graph Lottery Tickets: Graph Sparsity Matters, Bo Hui showed that topological masks transfer across different graph datasets because they capture generalized inductive biases for information flow rather than dataset-specific configurations.
- Functional Circuits: As explored by Enyan Zhang et al. in Instilling Inductive Biases with Subnetworks, transferring a sparse, functional sub-circuit often yields better generalization than transferring an entire pre-trained model. By isolating only the necessary logic for a subtask, the model avoids inheriting brittle heuristics or irrelevant features from the source domain.
By viewing sparsity not merely as a compression technique but as a method for isolating structural priors, this direction explains why the underlying connectivity patterns of neural networks can generalize as powerfully as the weights themselves.
Key Foundational Work
The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models (2020) — contributed 3 concept elements to this direction.
The Lottery Ticket Hypothesis for Object Recognition (2020) — contributed 5 concept elements to this direction.
The Lottery Ticket Hypothesis for Pre-trained BERT Networks (2020) — contributed 3 concept elements to this direction.