In traditional machine learning, we "teach" a model by adjusting its weights. However, a growing body of research suggests that for sufficiently large models, the training process might not be about changing weights at all, but rather about discovery.
The Strong Lottery Ticket Hypothesis (SLTH) posits that within a large, randomly initialized neural network, there exists a subnetwork that—when isolated by a binary mask—can match the performance of a fully trained model without a single weight update. Recent work has expanded this from theoretical curiosity to a practical toolkit for model compression, explainability, and task localization.
The Search for the Hidden Jackpot
If the solution already exists within the random weights of a model, the challenge shifts from optimization to search. Finding the right binary mask (which weights to keep and which to prune) is a daunting combinatorial problem.
Yuxin Zhang, Mingbao Lin, and their colleagues explored this in Lottery Jackpots Exist in Pre-trained Models. They found that pre-trained models contain "Jackpots"—highly sparse subnetworks that require no additional training. The key to finding them efficiently is smart initialization. By starting the mask search using magnitude-based pruning (removing the smallest weights first), researchers provide a high-quality inductive bias that drastically accelerates the discovery of these high-performing structures.
Scaling for Attention Mechanisms
While the SLTH was initially proven for simple networks, modern architectures like Transformers present unique challenges. In The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanisms, researchers including Daiki Chijiwa and Hikari Otsuka provided the first formal proof that these "strong" tickets exist within Multi-Head Attention (MHA).
Their work reveals that the "luck" of finding a subnetwork depends heavily on how the weights are initialized. They demonstrated that scaling the initialization of Query and Key weights differently from Value and Output weights—specifically by a factor related to the fourth root of the hidden dimension—optimizes the search. This asymmetric scaling aligns the random weight distribution with the theoretical requirements for approximating a trained attention function, making the "needle" much easier to find in the haystack.
Importance Scores and Robustness
The search for subnetworks isn't just about accuracy; it can also be used to bake in specific properties like adversarial robustness. In the HYDRA framework, Vikash Sehwag and Prateek Mittal treat pruning as an importance-score learning problem.
Instead of just looking at weights, they learn which connections are "important" for a robust objective. By initializing these importance scores proportionally to the magnitudes of pre-trained weights, they achieve massive compression (up to 99%) while maintaining robustness. This reinforces the idea that the pre-trained weights themselves serve as a powerful prior for where the most useful subnetworks reside.
Accelerating Discovery with Gradients
Beyond model compression, mask-based methods are used in Explainable AI (XAI) to identify which input features a model relies on. However, optimizing these masks is often slow.
Buelent Uendes and Shujian Yu introduced StartGrad in Start Smart: Leveraging Gradients For Enhancing Mask-based XAI Methods. By initializing the mask search with the absolute gradient of the model's prediction, they provide a "smart" starting point in regions of high sensitivity. This simple change in initialization reduces optimization time by over 6x, proving that even in the context of explainability, the model's own internal signals (like gradients) are the best guide for finding optimal masks.
Learnable Sparsity in Large Models
As models grow to the scale of LLMs, manual pruning criteria become insufficient. Gongfan Fang and Hongxu Yin addressed this with MaskLLM, which learns semi-structured sparsity patterns.
MaskLLM uses a Gumbel Softmax to learn the mask distribution, but it doesn't start from scratch. It initializes this learnable distribution using importance scores from one-shot pruning methods. This hybrid approach—using a heuristic to start and a learnable process to refine—allows for "lossless" sparsity on downstream tasks without ever updating the underlying model weights.
Why It Works: The Power of the Prior
The recurring theme across these papers is that initialization is destiny. Whether it is:
- Magnitude-based priors for finding Jackpots or robust subnetworks,
- Gradient-based priors for fast explainability, or
- Theoretically-scaled priors for attention mechanisms,
the core insight is the same: the weights of a large model (whether pre-trained or random) contain a vast library of potential functions. We do not need to "train" new functions into the model; we simply need to use the model's own internal geometry to find the mask that reveals the function we need. By starting the search in the right neighborhood, we turn a nearly impossible optimization problem into an efficient discovery process.