Abstract
Recent years have witnessed the great advances of deep neural networks (DNNs) in light field (LF) image super-resolution (SR). However, existing DNN-based LF image SR methods are developed on a single fixed degradation (e.g., bicubic downsampling), and thus cannot be applied to super-resolve real LF images with diverse degradation. In this article, we propose a simple yet effective method for real-world LF image SR. In our method, a practical LF degradation model is developed to formulate the degradation process of real LF images. Then, a convolutional neural network is designed to incorporate the degradation prior into the SR process. By training on LF images using our formulated degradation, our network can learn to modulate different degradation while incorporating both spatial and angular information in LF images. Extensive experiments on both synthetically degraded and real-world LF images demonstrate the effectiveness of our method. Compared with existing state-of-the-art single and LF image SR methods, our method achieves superior SR performance under a wide range of degradation, and generalizes better to real LF images.
Introduction
Light field (LF) cameras, such as those from Lytro or RayTrix, capture both the intensity and direction of light rays, enabling advanced applications like digital refocusing and depth estimation. However, these benefits often come at the cost of reduced spatial resolution. While deep learning has significantly improved super-resolution (SR) for standard images, applying these models to real-world LF data remains challenging.
The primary obstacles are the degradation gap and the information gap. Most existing SR models are trained on synthetic data with simple bicubic downsampling, failing to account for the lens blur and sensor noise inherent in real-world captures. Furthermore, it is difficult to effectively utilize degradation information (e.g., blur kernels) while maintaining the intricate geometric relationships between different LF views.
Methodology
To address these challenges, we propose LF-DMnet, which utilizes three key innovations:
- Practical LF Degradation Model: We model the low-resolution (LR) image formation as $I_{LR} = (I_{HR} \otimes \text{blur kernel}) \downarrow_{\text{sampling}} + \text{noise}$. This realistic simulation allows the network to learn from data that closely resembles real-world sensor output.
- Kernel Prior Embedding (KPE): The blur kernel and noise levels are transformed into a high-dimensional degradation representation vector, providing the network with explicit knowledge of the corruption it needs to reverse.
- Degradation-Modulating Block (DM-Block): Instead of static filters, we use DM-Conv layers where convolution weights are dynamically generated based on the degradation representation. This allows the network to adapt its processing—for instance, becoming more "smoothing" for noisy inputs or more "sharpening" for blurry ones.
Experimental Results
We evaluated LF-DMnet using a combination of synthetic datasets (HCInew, HCIold, STFgantry) for training and real-world datasets (EPFL, INRIA, STFlytro, RayTrix) for testing.
Quantitatively, LF-DMnet outperformed the previous state-of-the-art, DistgSSR, by up to 1.6 dB in PSNR on synthetically degraded benchmarks. Qualitatively, the model demonstrated exceptional generalization to real-world images. As shown in the comparison below, LF-DMnet produces significantly clearer text and sharper edges on real Lytro captures without the ringing artifacts or over-smoothing common in other methods like DASR or Real-ESRGAN.
Conclusion
LF-DMnet represents a significant step toward practical light field super-resolution. By dynamically modulating network weights based on degradation priors, it effectively bridges the gap between synthetic training and real-world application. While the current "non-blind" setting requires degradation parameters as input, future work will focus on "blind" LF SR, where the network automatically estimates these parameters from the light field structure itself.
Summary updated