Light Field View Synthesis via Aperture Flow and Propagation Confidence Map

Nan Meng, Kai Li, Jianzhuang Liu, Edmund Y. Lam

Abstract

This paper presents a learning-based approach to synthesize the view from an arbitrary camera position given a sparse set of images. A key challenge for this novel view synthesis arises from the reconstruction process, when the views from different input images may not be consistent due to obstruction in the light path. We overcome this by jointly modeling the epipolar property and occlusion in designing a convolutional neural network. We start by defining and computing the aperture disparity map, which approximates the parallax and measures the pixel-wise shift between two views. While this relates to free-space rendering and can fail near the object boundaries, we further develop a warping confidence map to address pixel occlusion in these challenging regions. The proposed method is evaluated on diverse real-world and synthetic light field scenes, and it shows better performance over several state-of-the-art techniques.

Introduction

Light field imaging captures both the intensity and direction of light rays, enabling post-capture refocusing and 3D reconstruction. However, hardware constraints often force a trade-off between spatial and angular resolution. To achieve high-resolution "dense" light fields, researchers typically capture a sparse set of images and synthesize the missing views.

Existing methods often struggle with two main issues: geometry-based methods fail in complex occlusion or semi-transparent regions where depth is difficult to estimate, while image-based rendering (IBR) requires dense sampling to avoid ghosting artifacts. This work introduces a "depth-free" but "geometry-aware" neural network that handles tricky object boundaries and occlusions by jointly modeling light field geometry and visibility.

Methodology

The proposed framework sidesteps explicit global depth estimation in favor of a three-stage pipeline that exploits the inherent structure of light fields.

1. Aperture Disparity Estimation

The network first estimates an Aperture Disparity Map (ADM). This leverages the "epipolar property"—the linear relationship between a camera's movement and the corresponding pixel shift. By estimating the shift between boundary views, the model analytically derives the shift for any intermediate viewpoint.

Light ray transmission and EPI patterns in free space

As shown in the figure above, the Epipolar Plane Image (EPI) exhibits a clear linear pattern where each line's slope corresponds to the scene depth. The ADM captures this "aperture flow" to warp input images to the target position.

2. Confidence Map Estimation for Occlusion Handling

In regions with occlusions, a pixel might be visible in one boundary view but blocked in another. To address this, the authors introduce Warping Confidence Maps (WCM). This learned weighting mechanism identifies which input image provides the "true" radiance for a target pixel.

EPI pattern modeling under occlusion

The figure illustrates how occluders disrupt the simple linear EPI lines. The WCM effectively "picks" the correct source from the boundary images, solving the occlusion problem without a full 3D reconstruction.

3. RefineNet with Alternating Filters

Initial warped results are refined through a 4D Convolutional Neural Network. To maintain efficiency, the authors use alternating filters (separating spatial and angular convolutions). This effectively exploits the high-dimensional redundancy of the light field without the massive memory overhead of standard 4D kernels.

Experimental Results

The model was evaluated on the Stanford Lytro Archive, HCI synthetic datasets, and Lytro Illum datasets.

  • Performance: The method consistently outperforms state-of-the-art models like Kalantari et al. and Soft3D. For example, on the Flowers1 scene, it achieved a PSNR of 34.64 dB.
  • Visual Quality: Results show significantly fewer artifacts around thin structures (like flower petals) and better consistency in synthesized EPIs.
  • Efficiency: Inference takes approximately 0.23 seconds per view, roughly 40x faster than previous optimization-based approaches.

Conclusion

By baking physical constraints—specifically the linear epipolar geometry—into the learning process, this method achieves robust and efficient view synthesis. While it currently relies on the linearity assumption of pinhole arrays and boundary image interpolation, it represents a significant step toward real-time, high-quality light field reconstruction.

Summary updated

Abstract

This paper presents a learning-based approach to synthesize the view from an arbitrary camera position given a sparse set of images. A key challenge for this novel view synthesis arises from the reconstruction process, when the views from different input images may not be consistent due to obstruction in the light path. We overcome this by jointly modeling the epipolar property and occlusion in designing a convolutional neural network. We start by defining and computing the aperture disparity map, which approximates the parallax and measures the pixel-wise shift between two views. While this relates to free-space rendering and can fail near the object boundaries, we further develop a warping confidence map to address pixel occlusion in these challenging regions. The proposed method is evaluated on diverse real-world and synthetic light field scenes, and it shows better performance over several state-of-the-art techniques.
BibTeX
@article{meng2021light,
  author = {Nan Meng and Kai Li and Jianzhuang Liu and Edmund Y. Lam},
  title = {{Light Field View Synthesis via Aperture Flow and Propagation Confidence Map}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=Pf3SrcU1Buk9},
}
Figure