Abstract
A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.
Introduction
Light fields (LF) capture the rich spatial and angular information of a scene, enabling advanced visual effects like post-capture refocusing and immersive 3D experiences. However, capturing a high-resolution, densely-sampled light field typically requires expensive camera arrays or slow, specialized gantries.
Computational reconstruction offers a solution by "filling in" missing viewpoints from sparse samples. Yet, existing methods face significant hurdles: they often require rigid, grid-like sampling patterns, struggle with the "ghosting" artifacts caused by large disparities between views, and suffer from the high computational cost of 4D convolutions. This paper introduces a framework that addresses these challenges through a geometry-aware, coarse-to-fine reconstruction strategy that remains efficient and flexible to irregular input patterns.
Proposed Architecture
The proposed network architecture consists of two primary stages designed to bridge the gap between sparse, irregular inputs and a dense, high-quality output.
1. Coarse SAI Synthesis Network
To handle irregular sampling and large disparities, this module explicitly models scene geometry using Plane-Sweep Volumes (PSVs). By encoding the relative positions of input cameras and target views into the data structure, the network can bridge large angular gaps. A key innovation here is the confidence-based blending strategy. Instead of simple averaging, the network predicts confidence maps for each warped input view, allowing it to intelligently select pixels from the most reliable viewpoints and effectively handle occlusions at depth boundaries.
2. Efficient LF Refinement Network
While the coarse stage synthesizes views independently, the refinement module recovers the global LF parallax structure. To maintain efficiency, the authors utilize Pseudo 4-D Convolutions, which use separable filters alternating between spatial and angular domains. This recovers consistency across views without the prohibitive cost of true 4D convolutions. The module employs residual learning to preserve high-frequency textures from the coarse stage.
Sampling Pattern Optimization
A secondary contribution of this work is the optimization of sparse sampling patterns. Using a deterministic annealing approach, the authors sought to identify which $K$ viewpoints provide the most information for reconstruction. They discovered a strong correlation between the "minimum distance" of a sampling pattern (the average distance from novel views to their nearest sampled neighbor) and the final reconstruction quality. This insight allows for the design of optimized sparse patterns that significantly outperform random or regular grids.
Results and Analysis
The model was evaluated on both synthetic (HCI) and real-world (Lytro Illum) datasets. On the HCI benchmark, which features challenging large disparities, the proposed method outperformed state-of-the-art models by up to 4 dB in PSNR.
Key findings include:
- Geometry-Awareness: Explicitly using PSVs and warping is essential for handling sparse inputs where "blind" regression typically fails.
- Flexibility: The network successfully reconstructs dense light fields from as few as 2 to 4 input views in arbitrary configurations.
- Downstream Utility: Reconstructed light fields were shown to improve the performance of existing depth estimation algorithms, proving the high fidelity of the synthesized parallax structures.
Conclusion
This work presents a flexible and efficient framework for light field reconstruction. By combining geometry-based synthesis with pseudo 4D refinement, the network achieves high-quality results from irregular sparse samples. Future work may involve scene-dependent sampling optimization and the integration of this reconstruction capability into light field compression pipelines.
Summary updated