Memory-Friendly Scalable Super-Resolution via Rewinding Lottery Ticket Hypothesis

Jin Lin, Xiaotong Luo, Ming Hong, Yanyun Qu, Yuan Xie, Zongze Wu

Abstract

Single Image Super-Resolution (SISR) models are typically fixed in size, posing challenges for deployment across diverse hardware with varying memory constraints. We present Memory-friendly Scalable Super-Resolution (MSSR), a framework that enables a single model to dynamically scale its parameters. By leveraging the Lottery Ticket Hypothesis and nested sub-networks, MSSR allows for significant parameter pruning (up to 94%) while maintaining high image quality. Our approach utilizes weight rewinding and stochastic self-distillation to ensure that sparse sub-networks perform competitively with their dense counterparts, providing a versatile solution for edge device deployment.

Introduction

The computational demands of high-performance SISR models often exceed the capabilities of edge devices. Traditional solutions involve either storing multiple models of different sizes or accepting the latency of reloading models when memory availability changes. MSSR addresses this by creating an unstructured scalable model where a single set of weights can be masked to generate various high-performing "winning ticket" sub-networks.

Overview of Model Scalability and Deployment in the MSSR framework.

Methodology

The MSSR framework operates in two primary stages: the Forward Stage for compression and the Backward Stage for expansion.

The Forward Stage: Model Compression

To identify optimal sparse sub-networks, the framework employs Iterative Magnitude Pruning (IMP) guided by the Lottery Ticket Hypothesis (LTH).

  • Weight Rewinding: Instead of standard fine-tuning, remaining weights are "rewound" to their values from an earlier training iteration, enabling sparse networks to match dense performance.
  • Nested Masks: Masks are designed to be nested (e.g., a 20% sparse model is a subset of a 40% sparse model). This allows a single weight matrix to support multiple scales simultaneously, minimizing storage requirements.
  • Stochastic Self-Distillation (SSD): To mitigate quality loss, the larger model iteration acts as a "Teacher" for the smaller "Student" iteration. Feature maps are randomly swapped during training to transfer high-frequency detail knowledge effectively.
The detailed pipeline of the Forward (Compression) and Backward (Expansion) stages.

The Backward Stage: Model Expansion

When additional memory becomes available, the model can "expand" by recovering previously pruned weights. These weights are then fine-tuned using the saved nested masks, allowing the model to regain capacity and improve image quality without the need for full retraining or reloading a separate model.

Experimental Results

We evaluated MSSR using the DIV2K training set and benchmarks including Set5, Set14, B100, and Urban100 across various backbones such as EDSR, RCAN, and RDN.

  • Extreme Sparsity: An EDSR model pruned by 94% (from 1.5M to 95K parameters) showed a negligible PSNR loss of only 0.22 dB.
  • Efficiency: At high sparsity levels, MSSR-pruned models outperformed dedicated lightweight architectures like SRCNN and FSRCNN while maintaining a smaller memory footprint.
  • Universal Applicability: The framework consistently improved performance across all tested architectures. Notably, in some cases like RCAN, the pruned and distilled model actually outperformed the original dense model at specific sparsity levels.

Discussion and Conclusion

MSSR demonstrates that "winning tickets" exist for image restoration tasks, not just classification. By enforcing nested masks, the framework achieves high memory efficiency without the overhead of storing multiple models. While the current implementation focuses on unstructured pruning—which may require specialized libraries for inference speedup—it sets a new standard for scalable super-resolution. Future work will explore structured pruning for broader hardware compatibility and application to video-based tasks.

Summary updated

Abstract

Scalable deep Super-Resolution (SR) models are increasingly in demand, whose memory can be customized and tuned to the computational recourse of the platform. The existing dynamic scalable SR methods are not memory-friendly enough because multi-scale models have to be saved with a fixed size for each model. Inspired by the success of Lottery Tickets Hypothesis (LTH) on image classification, we explore the existence of unstructured scalable SR deep models, that is, we find gradual shrinkage subnetworks of extreme sparsity named winning tickets. In this paper, we propose a Memory-friendly Scalable SR framework (MSSR). The advantage is that only a single scalable model covers multiple SR models with different sizes, instead of reloading SR models of different sizes. Concretely, MSSR consists of the forward and backward stages, the former for model compression and the latter for model expansion. In the forward stage, we take advantage of LTH with rewinding weights to progressively shrink the SR model and the pruning-out masks that form nested sets. Moreover, stochastic self-distillation (SSD) is conducted to boost the performance of sub-networks. By stochastically selecting multiple depths, the current model inputs the selected features into the corresponding parts in the larger model and improves the performance of the current model based on the feedback results of the larger model. In the backward stage, the smaller SR model could be expanded by recovering and fine-tuning the pruned parameters according to the pruning-out masks obtained in the forward. Extensive experiments show the effectiveness of MMSR. The smallest-scale sub-network could achieve the sparsity of 94% and outperforms the compared lightweight SR methods.
BibTeX
@inproceedings{lin2023memory,
  author = {Jin Lin and Xiaotong Luo and Ming Hong and Yanyun Qu and Yuan Xie and Zongze Wu},
  title = {{Memory-Friendly Scalable Super-Resolution via Rewinding Lottery Ticket Hypothesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01384},
}
Figure