Abstract
With the introduction of consumer light field cameras, light field imaging has recently become widespread. However, there is an inherent trade-off between the angular and spatial resolution, and thus, these cameras often sparsely sample in either spatial or angular domain. In this paper, we use machine learning to mitigate this trade-off. Specifically, we propose a novel learning-based approach to synthesize new views from a sparse set of input views. We build upon existing view synthesis techniques and break down the process into disparity and color estimation components. We use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. We show the performance of our approach using only four corner sub-aperture views from the light fields captured by the Lytro Illum camera. Experimental results show that our approach synthesizes high-quality images that are superior to the state-of-the-art techniques on a variety of challenging real-world scenes.
Introduction: The Resolution Tax
Light field cameras capture the 4D plenoptic function, recording both the intensity and direction of light. This rich data enables post-capture refocusing and perspective shifts. However, consumer devices like the Lytro Illum face a fundamental "resolution tax": the fixed sensor area must be shared between spatial resolution (pixels per view) and angular resolution (number of views). To achieve high spatial resolution, one must often settle for a sparse set of viewpoints.
Existing interpolation methods often fail at complex occlusion boundaries or produce "ghosting" artifacts. This paper introduces a deep learning framework designed to synthesize a dense grid of high-resolution views from just four corner sub-aperture images, effectively bypassing the hardware trade-off.
Proposed Method: Modular Deep Learning
Rather than a single black-box network, the authors propose a modular architecture that incorporates geometric priors. The pipeline consists of two main stages:
- Disparity Estimation: A CNN predicts a disparity map for the target novel view by analyzing features from the four input corner views. The system warps inputs across 100 disparity levels to calculate mean and standard deviation features.
- Color Prediction and Blending: The input views are "backward warped" to the target position using the predicted disparity. A second CNN then learns to blend these warped images, intelligently handling occlusions and correcting for minor errors in the disparity map.
A key innovation is the simultaneous training of both networks. By optimizing for the final image quality (color error) rather than just disparity accuracy, the system learns to produce "synthesis-friendly" disparities that work well even in textureless or ambiguous regions.
Experimental Results
The model was trained on a diverse dataset of 100 real-world light fields captured with a Lytro Illum. By training on real data, the network learned to be robust against the specific noise and optical distortions of consumer hardware. Quantitatively, the method achieved a PSNR of 37.50 dB, significantly outperforming traditional optimization-based baselines (e.g., 34.33 dB for Wanner and Goldluecke).
Qualitatively, the system excels at preserving sharp edges and complex structures, such as foliage, which typically pose significant challenges for view synthesis. While the system takes approximately 12.3 seconds per view on a mid-range GPU, it represents a massive speedup over previous learning-based competitors.
Conclusion
By combining geometric principles with the power of deep learning, this approach provides a robust solution to the angular-spatial resolution trade-off in light field imaging. The ability to synthesize high-quality views from sparse inputs suggests that future cameras could be designed with higher spatial resolution, relying on software to reconstruct the necessary angular detail. Future work includes optimizing for real-time performance and extending the model to handle arbitrary input patterns.
Summary updated