Abstract
This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation; the other one constructs another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations via the learned attention maps, leading to the final high-resolution LF image. Extensive experiments demonstrate the significant superiority of our approach over state-of-the-art ones. That is, our method not only improves the PSNR by more than 2 dB, but also preserves the LF structure much better. To the best of our knowledge, this is the first end-to-end deep learning method for reconstructing a high-resolution LF image with a hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and also be beneficial to LF data storage and transmission. The code is available at https://github.com/jingjin25/LFhybridSR-Fusion.
Introduction
Light Field (LF) imaging captures not just the intensity of light, but also the direction of incoming rays, enabling applications like post-capture refocusing and 3D reconstruction. However, there is a fundamental hardware trade-off: devices typically offer either high spatial resolution or high angular resolution, but rarely both in a compact form factor.
A promising solution is a hybrid imaging system, which uses a sparse grid of low-resolution (LR) sensors surrounding a single high-resolution (HR) central camera. The challenge lies in "spreading" the high-frequency textures from that single HR center view to all the LR side views while maintaining the correct 4D geometric consistency (the "LF structure"). Existing methods often rely on independent, multi-step pipelines that are prone to error accumulation and blurring.
Methodology
We propose an end-to-end architecture consisting of two parallel, complementary branches whose outputs are merged using a learned attention mechanism.
SR-Net (Spatial Consistency)
This branch progressively up-samples the LR views. It utilizes Spatial-Angular Separable (SAS) convolutions to efficiently process 4D data without the massive memory overhead of standard 4D convolutions. By alternating between 2D spatial and 2D angular convolutions, the model captures the 4D structure with significantly fewer parameters. It also incorporates a down-sampling branch that extracts features from the HR central view to guide the reconstruction.
Warp-Net (Texture Preservation)
This branch focuses on propagating pixels directly from the HR central view. It estimates a 4D disparity map from the LR views and uses it to warp the HR center view to all other viewpoints. This preserves delicate, high-frequency textures but can be prone to artifacts in occlusion zones or areas with inaccurate disparity estimation.
Attention-Guided Fusion
The model learns pixel-wise attention maps for both branches to adaptively blend the two intermediate results. It favors Warp-Net for sharp textures and SR-Net for regions where warping fails, such as occlusions or textureless areas, ensuring a high-fidelity final output.
Experiments and Results
The model was trained and evaluated on the synthetic HCI Dataset and real-world Lytro Dataset.
- Quantitative Performance: On the HCI dataset (4x scale), the proposed method achieved a PSNR of 38.69 dB, outperforming the next best method (CrossNet) by over 2.2 dB.
- Structural Fidelity: The method excels at preserving the Epipolar Plane Image (EPI) structure. In a valid light field, scene points form straight lines in the EPI; our approach maintains these lines much better than competitors, indicating superior geometric accuracy.
- Efficiency: Despite the dual-branch architecture, the model is significantly faster than traditional patch-based methods like PaSR.
Conclusion
This work introduces the first end-to-end deep learning framework for reconstructing high-resolution light fields from hybrid inputs. By combining learning-based super-resolution with geometry-based warping, the method achieves state-of-the-art performance in both numerical metrics and visual quality. Future research will focus on handling irregular camera geometries and applying this framework to light field data compression and transmission.
Summary updated