Learning Generalizable Light Field Networks from Few Images

Qian Li, Franck Multon, Adnane Boukhayma

Abstract

We explore a new strategy for few-shot novel view synthesis based on a neural light field representation. Given a target camera pose, an implicit neural network maps each ray to its target pixel's color directly. The network is conditioned on local ray features generated by coarse volumetric rendering from an explicit 3D feature volume. This volume is built from the input images using a 3D ConvNet. Our method achieves competitive performances on synthetic and real MVS data with respect to state-of-the-art neural radiance field based competition, while offering a 100 times faster rendering.

Introduction

Novel view synthesis—creating new images of a scene from a few starting photos—has been revolutionized by Neural Radiance Fields (NeRF). However, NeRF faces two major challenges: rendering speed and generalization. Rendering a single pixel in NeRF requires sampling hundreds of points along a ray, making real-time performance difficult. Furthermore, standard NeRFs are typically overfit to a single scene, requiring expensive retraining for new objects.

While Light Field Networks (LFNs) offer faster rendering by mapping rays directly to colors, they traditionally struggle to generalize to unseen scenes. This work bridges that gap by creating a light field network that is both fast and generalizable through local feature conditioning.

Methods

The proposed approach utilizes a hybrid pipeline combining explicit 3D volumes with implicit neural representations.

  1. 3D Feature Volume: A 2D U-Net and a 3D ConvNet extract a feature volume from each input image, aligned with the camera's view to encode geometry and appearance.
  2. Feature Resampling & Aggregation: Features are resampled into the target perspective. For multiple inputs, a learned confidence mechanism weighs the reliability of information from different views.
  3. Volumetric Feature Rendering: Instead of rendering colors directly, the model uses $\alpha$-compositing to render a low-resolution feature image, optimizing memory and speed.
  4. Neural Light Field: A Multi-Layer Perceptron (MLP) takes a ray (in Plücker coordinates) and its corresponding local feature to predict the final RGB color. Because the MLP is queried only once per pixel, rendering is significantly accelerated.
Overview of the proposed hybrid pipeline, showing the transition from input images to 3D feature volumes and finally the neural light field.

Experimental Results

The model was evaluated on the ShapeNet-V2 synthetic dataset (cars and chairs) and the DTU Dataset for real-world multi-view stereo.

  • Efficiency: On a Quadro RTX 5000 GPU, the method renders a $128 \times 128$ image in 0.06 seconds—a 100x speedup over PixelNeRF (6.23 seconds).
  • Quality and Generalization: The model achieves competitive PSNR and SSIM scores compared to state-of-the-art generalizable NeRF models. It demonstrates robust zero-shot generalization, synthesizing views of unseen objects from just 1 or 2 input images without per-scene fine-tuning.
Rendered novel views for chairs, cars, and real-world fruit scenes, demonstrating high-quality synthesis from sparse inputs.

Discussion and Limitations

While the method offers dramatic speed improvements, it faces challenges with high-frequency details due to the reduced resolution of the 3D feature volume. Additionally, the use of explicit dense volumes limits scalability to high-resolution or large-scale scenes. Future work will investigate sparse voxel or point cloud representations to overcome these memory constraints.

Summary updated

Abstract

We explore a new strategy for few-shot novel view synthesis based on a neural light field representation. Given a target camera pose, an implicit neural network maps each ray to its target pixel's color directly. The network is conditioned on local ray features generated by coarse volumetric rendering from an explicit 3D feature volume. This volume is built from the input images using a 3D ConvNet. Our method achieves competitive performances on synthetic and real MVS data with respect to state-of-the-art neural radiance field based competition, while offering a 100 times faster rendering.
BibTeX
@inproceedings{li2023learning,
  author = {Qian Li and Franck Multon and Adnane Boukhayma},
  title = {{Learning Generalizable Light Field Networks from Few Images}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096979},
}
Figure