Partial Search in a Frozen Network is Enough to Find a Strong Lottery Ticket

Hikari Otsuka, Daiki Chijiwa, Ángel López García-Arias, Yasuyuki Okoshi, Kazushi Kawamura, Thiem Van Chu, Daichi Fujiki, Susumu Takeuchi, Masato Motomura

Abstract

Randomly initialized dense networks contain subnetworks that achieve high accuracy without weight learning—strong lottery tickets (SLTs). Recently, Gadhikar et al. (2023) demonstrated that SLTs could also be found within a randomly pruned source network. This phenomenon can be exploited to further compress the small memory size required by SLTs. However, their method is limited to SLTs that are even sparser than the source, leading to worse accuracy due to unintentionally high sparsity. This paper proposes a method for reducing the SLT memory size without restricting the sparsity of the SLTs that can be found. A random subset of the initial weights is frozen by either permanently pruning them or locking them as a fixed part of the SLT, resulting in a smaller model size. Experimental results show that Edge-Popup finds SLTs with better accuracy-to-model size trade-off within frozen networks than within dense or randomly pruned source networks. In particular, freezing 70% of a ResNet on ImageNet provides 3.3× compression compared to the SLT found within a dense counterpart, raises accuracy by up to 14.12 points compared to the SLT found within a randomly pruned counterpart, and offers a better accuracy-model size trade-off than both.

Introduction

The Strong Lottery Ticket Hypothesis (SLTH) suggests that within a large, randomly initialized neural network, there exists a subnetwork that can achieve high accuracy without any weight training. Finding these "Strong Lottery Tickets" (SLTs) involves identifying the correct binary mask to apply to the random weights. While SLTs are attractive for memory-constrained environments—requiring only a random seed and a binary mask—the mask itself can still be quite large for deep architectures.

Previous attempts to compress this mask involved "pre-pruning" the random network before the search begins. However, this approach often forces the final subnetwork to be extremely sparse, which typically degrades accuracy—a phenomenon we call the "sparsity trap." We address this by introducing "Partial Freezing," a method that reduces the required mask size while maintaining the flexibility to find dense, high-performing subnetworks.

Methods: Partial Freezing

Our proposed technique divides the initial random weights into three categories before searching for the subnetwork:

  1. Pruned (Frozen): Randomly selected weights permanently set to zero.
  2. Locked (Frozen): Randomly selected weights permanently included in the subnetwork.
  3. Optimization Target: The remaining weights, which the search algorithm (e.g., Edge-Popup) can toggle to find the optimal subnetwork.
Conceptual overview of SLT search in dense, pruned, and frozen networks, alongside ImageNet performance results.

Because the patterns for pruned and locked weights are generated from a random seed, they do not need to be stored. Only the mask for the "Optimization Target" region requires storage, leading to significant memory savings. Crucially, "locking" weights allows the search to find denser subnetworks than pre-pruning alone, effectively bypassing the sparsity trap.

Experimental Results

We evaluated Partial Freezing across multiple datasets including CIFAR-10, ImageNet, and OGBN-Arxiv, using architectures ranging from ResNets to Graph Isomorphism Networks (GIN).

ImageNet Performance and Efficiency

On the ImageNet dataset, freezing 70% of a ResNet-50 model provided a 3.3× compression of the mask compared to searching a full dense network. As shown in Figure 1, this approach significantly outperformed simple pre-pruning (the "Pruned NN" baseline), maintaining a 14.12 point accuracy lead in some configurations. This demonstrates that the inclusion of locked weights is critical for maintaining performance in highly compressed regimes.

Freezing as a Regularizer

Surprisingly, searching in a partially frozen network often yielded higher accuracy than searching the full dense network. As illustrated in Figure 2, for architectures like Conv6 on CIFAR-10, freezing 40-60% of the network resulted in superior test accuracy compared to the 0% frozen baseline. We hypothesize that freezing reduces the search space, acting as a regularizer that helps the optimization algorithm avoid poor local optima.

Top-1 Test Accuracy vs. SLT Sparsity across different freezing ratios for various models and datasets.

Generalization to Other Domains

The method also showed exceptional results in non-vision tasks. On the OGBN-Arxiv graph node classification benchmark, a frozen SLT achieved 69.2% accuracy—matching the performance of a fully trained model—with a mask size of only 0.036 MB. Similarly, for ResNet-18 on CIFAR-10, we found a ticket with 89.8% accuracy using a mask of just 0.24 MB.

Discussion and Limitations

While Partial Freezing is highly effective for CNNs and GNNs, its performance on Vision Transformers (DeiT) was less impressive. Although it still outperformed sparse search methods, the overall accuracy remained well below that of trained models, indicating that SLT search in Transformers remains an open challenge. Furthermore, while the method is designed for specialized hardware like the "Hiddenite" engine, physical implementation and energy-efficiency benchmarks are left for future work.

Conclusion

Partial search in a frozen network offers a superior trade-off between accuracy and model size for Strong Lottery Tickets. By combining pruning with weight locking, we can achieve significant compression without sacrificing the density required for high performance, while simultaneously benefiting from the regularizing effects of a constrained search space.

Summary updated

Abstract

Randomly initialized dense networks contain subnetworks that achieve high accuracy without weight learning--strong lottery tickets (SLTs). Recently, Gadhikar et al. (2023) demonstrated that SLTs could also be found within a randomly pruned source network. This phenomenon can be exploited to further compress the small memory size required by SLTs. However, their method is limited to SLTs that are even sparser than the source, leading to worse accuracy due to unintentionally high sparsity. This paper proposes a method for reducing the SLT memory size without restricting the sparsity of the SLTs that can be found. A random subset of the initial weights is frozen by either permanently pruning them or locking them as a fixed part of the SLT, resulting in a smaller model size. Experimental results show that Edge-Popup (Ramanujan et al., 2020; Sreenivasan et al., 2022) finds SLTs with better accuracy-to-model size trade-off within frozen networks than within dense or randomly pruned source networks. In particular, freezing $70\%$ of a ResNet on ImageNet provides $3.3 \times$ compression compared to the SLT found within a dense counterpart, raises accuracy by up to $14.12$ points compared to the SLT found within a randomly pruned counterpart, and offers a better accuracy-model size trade-off than both.
BibTeX
@article{otsuka2025partial,
  author = {Hikari Otsuka and Daiki Chijiwa and Ángel López García-Arias and Yasuyuki Okoshi and Kazushi Kawamura and Thiem Van Chu and Daichi Fujiki and Susumu Takeuchi and Masato Motomura},
  title = {{Partial Search in a Frozen Network is Enough to Find a Strong Lottery Ticket}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2025},
  url = {https://openreview.net/forum?id=3WEGJ349jV},
}
Figure