Abstract
Single Image Super-Resolution (SISR) models are typically fixed in size, posing challenges for deployment across diverse hardware with varying memory constraints. We present Memory-friendly Scalable Super-Resolution (MSSR), a framework that enables a single model to dynamically scale its parameters. By leveraging the Lottery Ticket Hypothesis and nested sub-networks, MSSR allows for significant parameter pruning (up to 94%) while maintaining high image quality. Our approach utilizes weight rewinding and stochastic self-distillation to ensure that sparse sub-networks perform competitively with their dense counterparts, providing a versatile solution for edge device deployment.
Introduction
The computational demands of high-performance SISR models often exceed the capabilities of edge devices. Traditional solutions involve either storing multiple models of different sizes or accepting the latency of reloading models when memory availability changes. MSSR addresses this by creating an unstructured scalable model where a single set of weights can be masked to generate various high-performing "winning ticket" sub-networks.
Methodology
The MSSR framework operates in two primary stages: the Forward Stage for compression and the Backward Stage for expansion.
The Forward Stage: Model Compression
To identify optimal sparse sub-networks, the framework employs Iterative Magnitude Pruning (IMP) guided by the Lottery Ticket Hypothesis (LTH).
- Weight Rewinding: Instead of standard fine-tuning, remaining weights are "rewound" to their values from an earlier training iteration, enabling sparse networks to match dense performance.
- Nested Masks: Masks are designed to be nested (e.g., a 20% sparse model is a subset of a 40% sparse model). This allows a single weight matrix to support multiple scales simultaneously, minimizing storage requirements.
- Stochastic Self-Distillation (SSD): To mitigate quality loss, the larger model iteration acts as a "Teacher" for the smaller "Student" iteration. Feature maps are randomly swapped during training to transfer high-frequency detail knowledge effectively.
The Backward Stage: Model Expansion
When additional memory becomes available, the model can "expand" by recovering previously pruned weights. These weights are then fine-tuned using the saved nested masks, allowing the model to regain capacity and improve image quality without the need for full retraining or reloading a separate model.
Experimental Results
We evaluated MSSR using the DIV2K training set and benchmarks including Set5, Set14, B100, and Urban100 across various backbones such as EDSR, RCAN, and RDN.
- Extreme Sparsity: An EDSR model pruned by 94% (from 1.5M to 95K parameters) showed a negligible PSNR loss of only 0.22 dB.
- Efficiency: At high sparsity levels, MSSR-pruned models outperformed dedicated lightweight architectures like SRCNN and FSRCNN while maintaining a smaller memory footprint.
- Universal Applicability: The framework consistently improved performance across all tested architectures. Notably, in some cases like RCAN, the pruned and distilled model actually outperformed the original dense model at specific sparsity levels.
Discussion and Conclusion
MSSR demonstrates that "winning tickets" exist for image restoration tasks, not just classification. By enforcing nested masks, the framework achieves high memory efficiency without the overhead of storing multiple models. While the current implementation focuses on unstructured pruning—which may require specialized libraries for inference speedup—it sets a new standard for scalable super-resolution. Future work will explore structured pruning for broader hardware compatibility and application to video-based tasks.
Summary updated