Flexible, Fast and Accurate Densely-Sampled Light Field Reconstruction Network

Jing Jin, Junhui Hou, Jie Chen, Huanqiang Zeng, Sam Kwong, Jingyi Yu

Abstract

A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods.

Introduction

Light fields (LF) capture the rich spatial and angular information of a scene, enabling advanced visual effects like post-capture refocusing and immersive 3D experiences. However, capturing a high-resolution, densely-sampled light field typically requires expensive camera arrays or slow, specialized gantries.

Computational reconstruction offers a solution by "filling in" missing viewpoints from sparse samples. Yet, existing methods face significant hurdles: they often require rigid, grid-like sampling patterns, struggle with the "ghosting" artifacts caused by large disparities between views, and suffer from the high computational cost of 4D convolutions. This paper introduces a framework that addresses these challenges through a geometry-aware, coarse-to-fine reconstruction strategy that remains efficient and flexible to irregular input patterns.

Proposed Architecture

The proposed network architecture consists of two primary stages designed to bridge the gap between sparse, irregular inputs and a dense, high-quality output.

The proposed coarse-to-fine architecture consists of a Coarse SAI Synthesis Network and an Efficient LF Refinement Network.

1. Coarse SAI Synthesis Network

To handle irregular sampling and large disparities, this module explicitly models scene geometry using Plane-Sweep Volumes (PSVs). By encoding the relative positions of input cameras and target views into the data structure, the network can bridge large angular gaps. A key innovation here is the confidence-based blending strategy. Instead of simple averaging, the network predicts confidence maps for each warped input view, allowing it to intelligently select pixels from the most reliable viewpoints and effectively handle occlusions at depth boundaries.

2. Efficient LF Refinement Network

While the coarse stage synthesizes views independently, the refinement module recovers the global LF parallax structure. To maintain efficiency, the authors utilize Pseudo 4-D Convolutions, which use separable filters alternating between spatial and angular domains. This recovers consistency across views without the prohibitive cost of true 4D convolutions. The module employs residual learning to preserve high-frequency textures from the coarse stage.

Sampling Pattern Optimization

A secondary contribution of this work is the optimization of sparse sampling patterns. Using a deterministic annealing approach, the authors sought to identify which $K$ viewpoints provide the most information for reconstruction. They discovered a strong correlation between the "minimum distance" of a sampling pattern (the average distance from novel views to their nearest sampled neighbor) and the final reconstruction quality. This insight allows for the design of optimized sparse patterns that significantly outperform random or regular grids.

Results and Analysis

The model was evaluated on both synthetic (HCI) and real-world (Lytro Illum) datasets. On the HCI benchmark, which features challenging large disparities, the proposed method outperformed state-of-the-art models by up to 4 dB in PSNR.

Reconstruction quality (PSNR) exhibits a strong negative correlation with the minimum distance of the sampling pattern across different input counts.

Key findings include:

  • Geometry-Awareness: Explicitly using PSVs and warping is essential for handling sparse inputs where "blind" regression typically fails.
  • Flexibility: The network successfully reconstructs dense light fields from as few as 2 to 4 input views in arbitrary configurations.
  • Downstream Utility: Reconstructed light fields were shown to improve the performance of existing depth estimation algorithms, proving the high fidelity of the synthesized parallax structures.

Conclusion

This work presents a flexible and efficient framework for light field reconstruction. By combining geometry-based synthesis with pseudo 4D refinement, the network achieves high-quality results from irregular sparse samples. Future work may involve scene-dependent sampling optimization and the integration of this reconstruction capability into light field compression pipelines.

Summary updated

Abstract

A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods. In addition, we illustrate the benefits and advantages of the proposed approach when applied in various LF-based applications, including image-based rendering and depth estimation enhancement.
BibTeX
@article{jin2022flexible,
  author = {Jing Jin and Junhui Hou and Jie Chen and Huanqiang Zeng and Sam Kwong and Jingyi Yu},
  title = {{Flexible, Fast and Accurate Densely-Sampled Light Field Reconstruction Network}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2020.3026039},
}
Figure