Exploiting Raw Images for Real-Scene Super-Resolution

Xiangyu Xu, Yongrui Ma, Wenxiu Sun, Ming-Hsuan Yang

Abstract

Super-resolution is a fundamental problem in computer vision which aims to overcome the spatial limitation of camera sensors. While significant progress has been made in single image super-resolution, most algorithms only perform well on synthetic data, which limits their applications in real scenarios. In this paper, we study the problem of real-scene single image super-resolution to bridge the gap between synthetic data and real captured images. We focus on two issues of existing super-resolution algorithms: lack of realistic training data and insufficient utilization of visual information obtained from cameras. To address the first issue, we propose a method to generate more realistic training data by mimicking the imaging process of digital cameras. For the second issue, we develop a two-branch convolutional neural network to exploit the radiance information originally-recorded in raw images. In addition, we propose a dense channel-attention block for better image restoration as well as a learning-based guided filter network for effective color correction. Our model is able to generalize to different cameras without deliberately training on images from specific camera types. Extensive experiments demonstrate that the proposed algorithm can recover fine details and clear structures, and achieve high-quality results for single image super-resolution in real scenes.

Introduction

Most Single Image Super-Resolution (SISR) algorithms are trained on synthetic data created via simple bicubic downsampling. However, real-world camera pipelines are far more complex. When a camera captures an image, light hits the sensor to create raw data, which is then processed by an Image Signal Processor (ISP). The ISP performs non-linear operations such as noise reduction, color correction, and compression to produce a standard RGB image (e.g., JPEG). By the time an SR algorithm attempts to process this final RGB image, much of the original high-frequency detail has been lost or distorted.

The authors identify two key advantages to using raw data:

  1. Linearity: Restoration on raw data is mathematically cleaner because degradations like blur and noise are linear in that space.
  2. Bit Depth: Raw images typically offer 12 or 14 bits of precision, providing a richer foundation for detail recovery than the 8 bits found in standard JPEGs.

Methodology

The proposed framework addresses the limitations of synthetic training and information loss through two core innovations. First, a realistic data generation pipeline simulates the physics of a digital camera, applying defocus blur, motion blur, and heteroscedastic Gaussian noise directly to linear raw data.

Second, the authors introduce a two-branch convolutional neural network (CNN) architecture designed to decouple structural restoration from color science.

The proposed two-branch framework for real-scene super-resolution, separating structural restoration from color correction.

The Restoration Branch utilizes raw sensor data and a Dense Channel-Attention (DCA) block within an encoder-decoder structure to recover high-resolution structural details. The Color Correction Branch uses a low-resolution RGB image as a reference. It employs a learning-based guided filter to predict pixel-wise transformation matrices that map the restored linear structures to the final high-fidelity color space. This separation allows the model to generalize across different camera types without sensor-specific retraining.

Experimental Results

The model was trained on a curated subset of the MIT-Adobe 5K dataset and evaluated on a new real-scene dataset comprising 100 image pairs from diverse hardware, including Nikon, Sony, Canon, Leica, and iPhone 6s Plus.

Quantitatively, the proposed method achieved a PSNR of 31.06 dB on synthetic "blind" SR tests, significantly outperforming standard models like RDN (29.93 dB). Qualitatively, the model excels at recovering fine textures—such as text on signs or intricate fabric patterns—that typically appear as blurry artifacts in other methods.

Visual comparison of super-resolution results on real-world scenes. The proposed method (e) recovers sharper details compared to the input (b) and previous methods (c, d).

The results demonstrate that the model successfully generalizes to unseen hardware, proving the effectiveness of the two-branch approach in handling real-world photography challenges.

Conclusion

This research demonstrates that exploiting raw sensor data and realistic imaging-process simulations can bridge the performance gap between synthetic and real-world super-resolution. While the current model faces challenges with extreme noise levels, its ability to recover sharp, color-accurate details from diverse camera sources marks a significant advancement for practical image restoration. Future work will focus on integrating more explicit noise-reduction components to better handle low-light scenarios.

Summary updated

Abstract

Super-resolution is a fundamental problem in computer vision which aims to overcome the spatial limitation of camera sensors. While significant progress has been made in single image super-resolution, most algorithms only perform well on synthetic data, which limits their applications in real scenarios. In this paper, we study the problem of real-scene single image super-resolution to bridge the gap between synthetic data and real captured images. We focus on two issues of existing super-resolution algorithms: lack of realistic training data and insufficient utilization of visual information obtained from cameras. To address the first issue, we propose a method to generate more realistic training data by mimicking the imaging process of digital cameras. For the second issue, we develop a two-branch convolutional neural network to exploit the radiance information originally-recorded in raw images. In addition, we propose a dense channel-attention block for better image restoration as well as a learning-based guided filter network for effective color correction. Our model is able to generalize to different cameras without deliberately training on images from specific camera types. Extensive experiments demonstrate that the proposed algorithm can recover fine details and clear structures, and achieve high-quality results for single image super-resolution in real scenes.
BibTeX
@article{xu2022exploiting,
  author = {Xiangyu Xu and Yongrui Ma and Wenxiu Sun and Ming-Hsuan Yang},
  title = {{Exploiting Raw Images for Real-Scene Super-Resolution}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=rExD0FFb_C},
}
Figure