Learning a Degradation-Adaptive Network for Light Field Image Super-Resolution

Yingqian Wang, Zhengyu Liang, Longguang Wang, Jun-Gang Yang, Wei An, Yulan Guo

Abstract

Recent years have witnessed the great advances of deep neural networks (DNNs) in light field (LF) image super-resolution (SR). However, existing DNN-based LF image SR methods are developed on a single fixed degradation (e.g., bicubic downsampling), and thus cannot be applied to super-resolve real LF images with diverse degradations. In this paper, we propose the first method to handle LF image SR with multiple degradations. In our method, a practical LF degradation model that considers blur and noise is developed to approximate the degradation process of real LF images. Then, a degradation-adaptive network (LF-DAnet) is designed to incorporate the degradation prior into the SR process. By training on LF images with multiple synthetic degradations, our method can learn to adapt to different degradations while incorporating the spatial and angular information. Extensive experiments on both synthetically degraded and real-world LFs demonstrate the effectiveness of our method. Compared with existing state-of-the-art single and LF image SR methods, our method achieves superior SR performance under a wide range of degradations, and generalizes better to real LF images.

Introduction

Light field (LF) cameras are unique because they capture both the intensity and direction of light rays, allowing for advanced post-capture applications like refocusing and depth estimation. However, these sensors often produce low-resolution images due to the physical trade-off between spatial and angular sampling. While deep learning has significantly improved LF super-resolution (LF-SR), most existing models are "one-trick ponies"—they are trained exclusively on clean, idealized bicubic downsampling.

The camera imaging model and sensor grid illustrate how light field data is captured and structured.

In the real world, LF images from devices like Lytro or Raytrix suffer from complex, varying degradations: lens blur, sensor noise, and varying shot conditions. When standard LF-SR models meet these "dirty" real-world images, their performance plummets, often amplifying noise or producing blurry results. Furthermore, single-image super-resolution (SISR) methods designed for real-world data ignore the crucial 4D structure (angular information) of light fields, leading to inconsistent views or "flickering" in reconstructed light fields.

The LF-DAnet Framework

The researchers propose LF-DAnet, which treats degradation as a feature rather than an obstacle. The core innovation is the ability to adapt the network's behavior based on the specific blur and noise characteristics of the input image.

Practical LF Degradation Model

Instead of simple scaling, the authors formulate a model that accounts for isotropic Gaussian blur ($k$), downsampling ($\downarrow_\alpha$), and additive white Gaussian noise ($N$). This formulation better approximates the actual degradation process of real LF images compared to standard bicubic models.

Degradation-Adaptive Architecture

The network incorporates several key components to achieve this flexibility:

  1. Kernel Prior Embedding (KPE): This module takes the blur kernel width and noise level as inputs and uses a multi-layer perceptron (MLP) to transform these parameters into a compact "degradation representation."
  2. Degradation-Modulating Convolution (DM-Conv): This is the core mechanism. Unlike standard convolutions with fixed weights, DM-Conv uses weights that are dynamically generated based on the degradation representation. This allows the network to adapt its filtering strategy on the fly depending on how blurry or noisy the input is.
  3. Spatial-Angular Disentangling: The network incorporates modified "Distg-Blocks" to separately process spatial and angular information, ensuring that the final high-resolution output maintains the geometric integrity of the light field.

Experimental Results

The authors utilized a mix of synthetic and real-world datasets for evaluation, including HCInew, HCIold, STFgantry, and real captures from EPFL, INRIA, and STFlytro.

Visual Fidelity and Performance

On synthetic benchmarks with heavy noise and blur, LF-DAnet outperformed the previous best LF-SR method (DistgSSR) by over 5 dB in PSNR. Despite the dynamic weight generation, the model maintains a moderate parameter count (3.80M) and fast inference times, making it efficient for practical use.

Visual comparison of SR results on real-world LF images. LF-DAnet (Ours) produces significantly sharper text and finer details compared to existing methods.

As shown in the comparison above, on real Lytro images, LF-DAnet produced significantly sharper text and finer details (e.g., on the "ISO Chart" and "Hot Chili Sauce" labels) compared to both general SR and specialized LF-SR competitors, which often left the images looking "muddy" or artifact-heavy.

Conclusion

LF-DAnet represents a significant advance in making light field super-resolution applicable to real-world photography. By conditioning the convolution kernels on noise and blur levels, the model can act as a sharpener for blurry images and a denoiser for grainy ones simultaneously. While the current method is non-blind—requiring degradation parameters as input—it provides a robust foundation for future blind LF-SR research that can automatically estimate these parameters from raw data.

Summary updated

Abstract

Recent years have witnessed the great advances of deep neural networks (DNNs) in light field (LF) image super-resolution (SR). However, existing DNN-based LF image SR methods are developed on a single fixed degradation (e.g., bicubic downsampling), and thus cannot be applied to super-resolve real LF images with diverse degradations. In this paper, we propose the first method to handle LF image SR with multiple degradations. In our method, a practical LF degradation model that considers blur and noise is developed to approximate the degradation process of real LF images. Then, a degradation-adaptive network (LF-DAnet) is designed to incorporate the degradation prior into the SR process. By training on LF images with multiple synthetic degradations, our method can learn to adapt to different degradations while incorporating the spatial and angular information. Extensive experiments on both synthetically degraded and real-world LFs demonstrate the effectiveness of our method. Compared with existing state-of-the-art single and LF image SR methods, our method achieves superior SR performance under a wide range of degradations, and generalizes better to real LF images. Codes and models are available at https://github.com/YingqianWang/LF-DAnet.
BibTeX
@article{wang2024learning,
  author = {Yingqian Wang and Zhengyu Liang and Longguang Wang and Jun-Gang Yang and Wei An and Yulan Guo},
  title = {{Learning a Degradation-Adaptive Network for Light Field Image Super-Resolution}},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  year = {2024},
  doi = {10.1109/tnnls.2024.3378420},
}
Figure