Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses

Jing Jin, Junhui Hou, Jie Chen, Sam Kwong, Jingyi Yu

Abstract: This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. Extensive experiments demonstrate the significant superiority of our approach over state-of-the-art ones. That is, our method not only improves the PSNR by more than 2 dB, but also preserves the LF structure much better. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and also be beneficial to LF data storage and transmission.

Introduction

Light Field (LF) imaging is a powerful modality that captures both the intensity and the direction of light rays, enabling advanced applications such as post-capture refocusing and 3D scene reconstruction. However, commercial LF cameras (like the Lytro) often suffer from a "resolution tax," where spatial resolution is sacrificed to capture angular information, resulting in relatively blurry images.

A promising solution is the hybrid imaging system: a sparse grid of inexpensive, low-resolution (LR) cameras surrounding a single high-resolution (HR) camera. The primary challenge in this setup is fusion: how to accurately propagate sharp textures from the central HR camera to the peripheral LR views without introducing artifacts or violating the underlying 4D geometry (parallax) of the scene. Existing methods often rely on multi-step processes prone to error accumulation. This paper introduces LFhybridSR-Fusion, the first end-to-end deep learning framework designed to solve this problem.

Methods

The proposed architecture processes the hybrid input through two parallel, complementary branches that address the trade-off between geometric consistency and texture sharpness.

Overview of the LFhybridSR-Fusion architecture, showing the dual-branch structure and attention-guided fusion.
  1. SR-Net (The Consistency Branch): This module focuses on maintaining spatial and angular consistency. It utilizes a cascaded hourglass structure to progressively up-sample the LR side views. To manage the computational complexity of 4D data, it employs Spatial-Angular Separable (SAS) convolutions, which alternate between processing spatial pixels and angular viewpoints. It uses features from the HR center view as a guide for the up-sampling process.
  2. Warp-Net (The Texture Branch): This module focuses on preserving high-frequency details. It estimates a 4D disparity map from the LR views and uses it to warp the HR center view's pixels into the side-view positions. This ensures the sharpest available textures are utilized, though it may struggle in occluded or textureless regions.
  3. Attention-Guided Fusion: The core innovation lies in the learned pixel-wise attention maps. Instead of simple averaging, the network identifies which branch performed better at every pixel. Typically, SR-Net is weighted more heavily in occluded or smooth areas, while Warp-Net dominates on sharp edges and complex textures.

Key Insights

  • Complementary Strengths: CNN-based super-resolution (SR-Net) excels at global structure but can be blurry, while warping (Warp-Net) provides sharp pixels but can create "ghosting" at edges. The attention mechanism acts as a smart judge to combine the best of both.
  • Geometric-Aware Augmentation: The authors developed a strategy that flips both spatial and angular dimensions simultaneously, ensuring the "physics" of the light field remains intact during training.
  • Efficiency: By separating 4D convolutions into spatial and angular components (SAS-Conv), the network remains deep enough to learn complex features without excessive memory requirements.

Key Results

The model was evaluated on the HCI Dataset (synthetic) and the Lytro Dataset (real-world). The results demonstrate a significant quantitative and qualitative leap over previous state-of-the-art (SOTA) methods like CrossNet and PaSR.

Qualitative comparison of 4x and 8x super-resolution results on HCI and Lytro datasets.
  • Quantitative Performance: The method improves PSNR by more than 2 dB across benchmarks, a substantial margin in image restoration tasks.
  • Structural Fidelity: Visualization of Epipolar Plane Images (EPIs) shows that the method preserves straight, clean lines of parallax, avoiding the distortions common in other methods.
  • Visual Sharpness: Fine textures, such as small text on labels, are clearly legible in the proposed method's output while remaining blurred in competing approaches.

Limitations and Future Work

While highly effective, the current study uses simulated hybrid data where cameras are perfectly aligned. Real-world physical rigs would require addressing sensor misalignments and color inconsistencies. Additionally, the current architecture assumes a regular 2D grid; future iterations could extend this to arbitrary, off-grid camera positions. Despite these challenges, the framework represents a major step toward lower-cost, high-resolution light field acquisition.

Summary updated

Abstract

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. Extensive experiments demonstrate the significant superiority of our approach over state-of-the-art ones. That is, our method not only improves the PSNR by more than 2 dB, but also preserves the LF structure much better. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and also be beneficial to LF data storage and transmission. The code is available at https://github.com/jingjin25/LFhybridSR-Fusion.
BibTeX
@inproceedings{jin2020light,
  author = {Jing Jin and Junhui Hou and Jie Chen and Sam Kwong and Jingyi Yu},
  title = {{Light Field Super-resolution via Attention-Guided Fusion of Hybrid Lenses}},
  booktitle = {ACM Multimedia},
  year = {2020},
  doi = {10.1145/3394171.3413585},
}
Figure