Functional Subnetworks in Randomly Weighted Neural Networks

This direction is merged into another direction: subnetwork discovery. Its last version stays readable and citable here.

Version 1 of 145 papers55 conceptsfrozen 2026-10-04

In traditional deep learning, we initialize a network with random weights and then use gradient descent to painstakingly nudge those values until the model performs a task. However, a growing body of research suggests a startling alternative: the weights you need might already be there.

The Strong Lottery Ticket Hypothesis (SLTH) proposes that a sufficiently large, randomly initialized neural network contains a subnetwork that can achieve high accuracy without any weight updates. Instead of "learning" weights, we simply need to "unhide" the right connectivity pattern.

From Training Weights to Searching Masks

The shift in perspective began with the work of Vivek Ramanujan, Mitchell Wortsman, and Mohammad Rastegari in What’s Hidden in a Randomly Weighted Neural Network?. They introduced the edge-popup algorithm, which treats the weights as fixed and instead learns a binary mask. By selecting the "right" random weights, they found a subnetwork within a random Wide ResNet-50 that matched the performance of a fully trained ResNet-34 on ImageNet.

This is fundamentally different from the original Lottery Ticket Hypothesis. While the original hypothesis suggests that certain sparse subnetworks are particularly well-suited for training, the "strong" version suggests that these subnetworks are already functional at initialization.

The Intuition: Combinatorial Abundance

Why should a random mess of weights contain a high-performing model? The answer lies in the combinatorial explosion of possible subnetworks.

Think of a large neural network as a massive library of random features. If the network is wide enough, the probability that some subset of those features can be combined to approximate a target function approaches certainty. Emanuele Natale and colleagues formalized this in On the Sparsity of the Strong Lottery Ticket Hypothesis by framing the search as a Random Fixed-Size Subset Sum (RFSS) problem. They proved that in a sufficiently large random network, you can always find a sparse subnetwork to approximate a target function by simply picking the right combination of random values.

This principle isn't limited to standard architectures. Rebekka Burkholz demonstrated in Convolutional and Residual Networks Provably Contain Lottery Tickets that this property is inherent to the convolutions and skip connections used in modern ResNets. Furthermore, Burkholz showed that this phenomenon holds across Most Activation Functions, including Sigmoid and Tanh, provided the network is slightly deeper than the target model.

Scaling to Complex Architectures

The existence of these "strong" tickets has been verified across increasingly complex domains:

Are Winning Tickets Rare?

Initially, these subnetworks were viewed as "rare gems" (Kartik Sreenivasan et al., Rare Gems: Finding Lottery Tickets at Initialization). However, recent evidence suggests they might be ubiquitous.

The Dual Lottery Ticket Hypothesis, proposed by Huan Wang in Dual Lottery Ticket Hypothesis, suggests that any random subnetwork can be transformed into a winning ticket with the right conditioning. Taking this further, Md Kowsher introduced the Universal Winning Slice Hypothesis in SliceFine, arguing that in pretrained models, almost any randomly selected block of weights can be effectively adapted to a downstream task.

Why This Matters

If we can find functional models through selection rather than optimization, we unlock new ways to build AI:

  1. Extreme Efficiency: Methods like Multi-Prize Lottery Tickets by James Diffenderfer find binary subnetworks that are highly accurate but require far less memory than full-precision models.
  2. Hardware Friendliness: Maxwell Mbabilla Aladago showed in Slot Machines that we can replace continuous weights with a small "menu" of random values, significantly simplifying the underlying hardware requirements.
  3. No-Training Representations: For tasks like video compression, Hee Min Choi demonstrated in Is Overfitting Necessary for Implicit Video Representation? that representing a signal as a subnetwork of a random model can outperform traditional training-heavy approaches.

By treating neural network weights as a fixed resource to be indexed rather than a set of variables to be tuned, this research direction suggests that the "intelligence" of a model may reside as much in its connectivity as in its parameter values.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_functional_subnetworks_in_randomly_weigh_e13c2586,
  title = {Functional Subnetworks in Randomly Weighted Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-subnetworks-in-randomly-weighted-neural-networks/rdr_e13c2586251b4694ba05017d6f259700},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_functional_subnetworks_in_randomly_weigh_e13c2586_v1,
  title = {Functional Subnetworks in Randomly Weighted Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/functional-subnetworks-in-randomly-weighted-neural-networks/rdr_e13c2586251b4694ba05017d6f259700/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}