Light Field View Synthesis via Aperture Disparity and Warping Confidence Map

Nan Meng, Kai Li, Jianzhuang Liu, Edmund Y. Lam

TL;DR

This paper introduces a learning-based framework for light field view synthesis that sidesteps explicit depth estimation. By jointly modeling the linear geometry of light fields through Aperture Disparity Maps (ADM) and handling occlusions with Warping Confidence Maps (WCM), the authors achieve state-of-the-art reconstruction quality that is up to 40x faster than previous optimization-based methods.

What Problem Does This Paper Address?

Light field imaging captures both the intensity and direction of light rays, allowing for post-capture refocusing and 3D reconstruction. However, there is a fundamental trade-off between spatial and angular resolution. To get high-resolution "dense" light fields, researchers often capture a sparse set of images and try to synthesize the missing views.

Existing methods usually fall into two traps:

  1. Geometry-based methods struggle with complex occlusions or semi-transparent surfaces where depth is hard to calculate.
  2. Image-based rendering (IBR) often requires too many input samples to avoid "ghosting" or aliasing artifacts.

The authors aim to bridge this gap by creating a "depth-free" but "geometry-aware" neural network that can handle the tricky regions around object boundaries where occlusions occur.

Methods

The proposed pipeline consists of three main stages:

  1. Aperture Disparity Estimation: Instead of calculating a global depth map, the network estimates an Aperture Disparity Map (ADM). This exploits the "epipolar property" of light fields—the fact that a pixel's shift between views is linearly related to the camera's movement. By estimating the shift between boundary views, the model can analytically derive the shift for any intermediate viewpoint.
  2. Confidence Map Estimation: To handle occlusions (where a pixel is visible in one input view but blocked in another), the authors introduce Warping Confidence Maps (WCM). This is a learned weighting mechanism that tells the system which input image provides the "true" radiance for a specific target pixel.
  3. RefineNet: The initial warped results are passed through a 4D Convolutional Neural Network. To keep the computation manageable, they use alternating filters (separating spatial and angular convolutions), which effectively exploits the high-dimensional redundancy of the light field without the massive memory overhead of standard 4D kernels.

Datasets

The model was trained and evaluated on a mix of synthetic and real-world data:

  • Stanford Lytro Archive: 100 real-world scenes used for training.
  • HCI Datasets: Synthetic scenes (e.g., Buddha, Dino) generated via Blender, providing perfect ground truth for evaluation.
  • Lytro Illum Datasets: Publicly available real-world datasets including Flower, Cars, and Seahorse to test generalization.

Key Insights and Ideas

  • The Boundary Property: A key insight is that an occluded pixel in a synthesized view almost always appears in either the leftmost or the rightmost boundary image. By using WCMs to "pick" the correct boundary source, the model elegantly solves the occlusion problem without needing a perfect 3D model of the scene.
  • Linearity as a Prior: By forcing the network to learn a disparity map that follows the linear epipolar geometry, the authors bake physical constraints into the learning process, making it more robust than "black-box" image-to-image translation models.
  • Efficiency through Alternating Filters: 4D convolutions are notoriously heavy. The use of alternating 3x3x1x1 and 1x1x3x3 filters allows the network to "see" across both the image plane and the camera grid efficiently.

Key Results

  • Quantitative: The method consistently outperformed state-of-the-art models like Kalantari et al. and Soft3D. For example, on the Flowers1 scene, it achieved a PSNR of 34.64 dB compared to Kalantari's 33.32 dB.
  • Qualitative: The results show significantly fewer artifacts around thin structures (like flower petals or tree branches) and better consistency in the Epipolar Plane Images (EPIs), which represent the "slope" of light rays.
  • Speed: The inference time is approximately 0.23 seconds per synthesized view, which is roughly 40x faster than the optimization-heavy approach by Kalantari et al. (~9s).

Limitations and Future Work

  • Linearity Assumption: The ADM relies on a linear relationship between viewpoint and pixel shift. While true for ideal pinhole arrays, extreme lens distortion in some hardware might violate this assumption.
  • Boundary Reliance: The current model is designed to interpolate between boundary images. Extending this to extrapolate views outside the initial camera grid or to handle unstructured camera "swarms" remains a challenge.
  • RefineNet Complexity: While alternating filters help, 4D processing is still more demanding than 2D, potentially limiting real-time applications on mobile hardware.

Summary updated

Abstract

This paper presents a learning-based approach to synthesize the view from an arbitrary camera position given a sparse set of images. A key challenge for this novel view synthesis arises from the reconstruction process, when the views from different input images may not be consistent due to obstruction in the light path. We overcome this by jointly modeling the epipolar property and occlusion in designing a convolutional neural network. We start by defining and computing the aperture disparity map, which approximates the parallax and measures the pixel-wise shift between two views. While this relates to free-space rendering and can fail near the object boundaries, we further develop a warping confidence map to address pixel occlusion in these challenging regions. The proposed method is evaluated on diverse real-world and synthetic light field scenes, and it shows better performance over several state-of-the-art techniques.
BibTeX
@article{meng2021light,
  author = {Nan Meng and Kai Li and Jianzhuang Liu and Edmund Y. Lam},
  title = {{Light Field View Synthesis via Aperture Disparity and Warping Confidence Map}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=Pf3SrcU1Buk9},
}

Loading related content...

Figure