Abstract: Deep networks have achieved great success in image rescaling (IR) task that seeks to learn the optimal downscaled representations, i.e., low-resolution (LR) images, to reconstruct the original high-resolution (HR) images. Compared with super-resolution methods that consider a fixed downscaling scheme, e.g., bicubic, IR often achieves significantly better reconstruction performance thanks to the learned downscaled representations. This highlights the importance of a good downscaled representation in image reconstruction tasks. Existing IR methods mainly learn the downscaled representation by jointly optimizing the downscaling and upscaling models. Unlike them, we seek to improve the downscaled representation through a different and more direct way: optimizing the downscaled image itself instead of the down-/upscaling models. Specifically, we propose a collaborative downscaling scheme that directly generates the collaborative LR examples by descending the gradient w.r.t. the reconstruction loss on them to benefit the IR process. Furthermore, since LR images are downscaled from the corresponding HR images, one can also improve the downscaled representation if we have a better representation in the HR domain. Inspired by this, we propose a Hierarchical Collaborative Downscaling (HCD) method that performs gradient descent in both HR and LR domains to improve the downscaled representations. Extensive experiments show that our HCD significantly improves the reconstruction performance both quantitatively and qualitatively. Moreover, we also highlight the flexibility of our HCD since it can generalize well across diverse IR models.
1. Introduction
Image Rescaling (IR) is the task of shrinking a high-resolution (HR) image into a low-resolution (LR) version for storage or transmission, then reconstructing it to its original size. Unlike standard Super-Resolution, where the downscaling method is typically fixed (e.g., bicubic), IR allows for learning an optimal downscaling scheme to facilitate better reconstruction. Current state-of-the-art approaches focus on optimizing model parameters through joint training of downscaling and upscaling networks. However, this paper identifies a missed opportunity: even with optimized models, the specific pixel values in the LR image may not be optimal for reconstruction. We propose directly optimizing the image representation itself to assist the upscaler.
2. Methodology
2.1 Collaborative Downscaling Concept
The core innovation is Hierarchical Collaborative Downscaling (HCD), which draws a parallel to adversarial attacks. While adversarial attacks use gradient ascent to find pixel perturbations that degrade model performance, HCD uses gradient descent to find "collaborative" perturbations that minimize reconstruction loss.
2.2 Hierarchical Optimization
HCD performs optimization across two levels to refine the downscaled representation:
- Collaborative HR: The original HR image is slightly modified so that its downscaled counterpart contains more useful information for the upscaler.
- Collaborative LR: Once the LR image is generated, its pixels are further refined via gradient descent.
This bi-level optimization is formulated as a process that alternates between the HR and LR domains. Crucially, this optimization occurs during the downscaling (encoding) phase, meaning the upscaling (decoding) phase remains a fast, single forward pass.
3. Experimental Evaluation
3.1 Datasets and Baselines
The method was evaluated on standard benchmarks including DIV2K (validation), Set5, Set14, BSD100, and Urban100. HCD was integrated with leading IR models such as IRN (Invertible Rescaling Network) and HCFlow.
3.2 Key Results
- Quantitative Performance: On the Set5 dataset at 4× scale, HCD improved the PSNR of the HCFlow baseline by 0.7 dB, a significant margin in image restoration tasks.
- Qualitative Quality: Reconstructed images showed noticeably sharper edges and fewer artifacts (such as ringing or blurriness) compared to baselines. Visualization indicates that HCD focuses perturbations on high-frequency areas like textures and contours.
- Model Agnostic: HCD consistently improved performance across different base models, demonstrating its flexibility as a plug-and-play enhancement.
4. Discussion and Limitations
HCD demonstrates that improving the input data representation can be as effective as architectural innovation. Since perturbations are baked into the LR image at the source, end-users receive a quality boost without additional inference overhead.
However, the iterative gradient descent required during downscaling (typically 15 iterations) increases encoding latency, making it less suitable for real-time video applications. Additionally, collaborative examples are optimized for specific upscaling models; using a different upscaler may diminish the benefits. Future work may explore interactions with standard compression formats like JPEG.
5. Conclusion
By optimizing the downscaled representation directly through hierarchical gradient descent, HCD significantly boosts image rescaling performance. This approach highlights the importance of data-centric optimization in deep learning-based image restoration.
Summary updated