Empowering Resampling Operation for Ultra-High-Definition Image Enhancement with Model-Aware Guidance

Wei Yu, Jie Huang, Bing Li, Kaiwen Zheng, Qi Zhu, Man Zhou, Feng Zhao

Abstract

Ultra-High-Definition (UHD) image enhancement is computationally demanding, necessitating a "Downsampling-Enhancement-Upsampling" paradigm. However, traditional resampling methods are independent of the enhancement model, leading to the loss of critical high-frequency details. This paper presents Learning Model-Aware Resampling (LMAR), a framework that correlates the resizer with a pre-trained enhancer. By utilizing Implicit Neural Representations and steganographic techniques, LMAR embeds model-specific compensatory information into downscaled images. Experimental results on 4K benchmarks demonstrate significant quality improvements, achieving up to 1.2dB PSNR gain without retraining the enhancement backbone.

Introduction

Processing 4K and 8K images directly with deep learning models often exceeds memory and computational limits. The standard industry solution involves downsampling the image, applying an enhancement model at a lower resolution, and then upsampling the result. A significant drawback of this approach is that standard resampling operators (e.g., Bicubic) are "blind" to the downstream enhancement model. They discard information indiscriminately, depriving the enhancer of specific cues needed for high-fidelity reconstruction.

Standard resampling methods lack correlation with the downstream enhancement model, leading to suboptimal feature preservation.

Learning Model-Aware Resampling (LMAR)

To bridge the gap between resampling and enhancement, we propose LMAR. This method "empowers" existing interpolation operators by making them aware of the specific requirements of a pre-trained, frozen enhancement model.

Compensatory Kernel Estimation (CKE)

LMAR employs an Implicit Neural Representation (INR) to predict image-specific convolution kernels. Unlike fixed mathematical formulas, these kernels are learned to capture "compensatory information"—the specific details that the enhancement model requires to produce a high-quality UHD output.

Steganographic Resampling (SR)

The estimated compensatory information is embedded into the downscaled image using a steganographic approach. This creates a "model-favored" low-resolution input. To the human eye, the image appears normal, but it contains hidden features that the enhancement model can exploit for better restoration.

The proposed LMAR framework correlates the downsampling and upsampling stages with the enhancement model to preserve task-specific information.

Experimental Evaluation

The effectiveness of LMAR was evaluated on UHD low-light enhancement datasets, including UHD-LOL4K and 4KIL.

Quantitative and Qualitative Results

LMAR consistently outperforms standard resampling across various scales (x1.5 to x5). On the UHD-LOL4K dataset, it achieved a PSNR improvement of up to 1.2dB at a x1.8 scale. The method is versatile, showing gains across different architectures such as CNNs and Transformers (e.g., Restormer). Qualitatively, LMAR produces sharper edges and superior global illumination compared to "blind" downsampling methods, which often result in blurriness or artifacts.

Efficiency and Compatibility

A key advantage of LMAR is its compatibility with frozen pre-trained models. It requires no retraining of the heavy enhancement backbone, only the lightweight LMAR module. The module adds minimal overhead, increasing model size by a small fraction (e.g., from 1.009MB to 1.314MB) with negligible impact on inference time.

Conclusion

LMAR addresses the fundamental disconnect in the "Downsampling-Enhancement-Upsampling" pipeline for UHD images. By making the resampling process model-aware, it preserves critical features that standard methods discard. This approach offers a practical and efficient way to boost the performance of existing UHD enhancement models without the need for expensive retraining.

Summary updated

Abstract

Image enhancement algorithms have made remarkable advancements in recent years, but directly applying them to Ultra-high-definition (UHD) images presents intractable computational overheads. Therefore, previous straightforward solutions employ resampling techniques to reduce the resolution by adopting a “Downsampling-Enhancement-Upsampling” processing paradigm. However, this paradigm disentangles the resampling operators and inner enhancement algorithms, which results in the loss of information that is favored by the model, further leading to sub-optimal outcomes. In this paper, we propose a novel method of Learning Model-Aware Resampling (LMAR), which learns to customize resampling by extracting model-aware information from the UHD input image, under the guidance of model knowledge. Specifically, our method consists of two core designs, namely compensatory kernel estimation and steganographic resampling. At the first stage, we dynamically predict compensatory kernels tailored to the specific input and resampling scales. At the second stage, the image-wise compensatory information is derived with the compensatory kernels and embedded into the rescaled input images. This promotes the representation of the newly derived downscaled inputs to be more consistent with the full-resolution UHD inputs, as perceived by the model. Our LMAR enables model-aware and model-favored resampling while maintaining compatibility with existing resampling operators. Extensive experiments on multiple UHD image enhancement datasets and different backbones have shown consistent performance gains after correlating resizer and enhancer; e.g., up to 1.2dB PSNR gain for ×1.8 resampling scale on UHD-LOL4K. The code is available at https://github.com/YPatrickW/LMAR.
BibTeX
@inproceedings{yu2024empowering,
  author = {Wei Yu and Jie Huang and Bing Li and Kaiwen Zheng and Qi Zhu and Man Zhou and Feng Zhao},
  title = {{Empowering Resampling Operation for Ultra-High-Definition Image Enhancement with Model-Aware Guidance}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02430},
}

Loading related content...

Figure