Fast and Explicit Neural View Synthesis

Pengsheng Guo, Miguel Ángel Bautista, Alex Colburn, Liang Yang, Daniel Ulbricht, Joshua M. Susskind, Qi Shan

Abstract

We study the problem of novel view synthesis from sparse source observations of a scene comprised of 3D objects. We propose a simple yet effective approach that is neither continuous nor implicit, challenging recent trends on view synthesis. Our approach explicitly encodes observations into a volumetric representation that enables amortized rendering. We demonstrate that although continuous radiance field representations have gained a lot of attention due to their expressive power, our simple approach obtains comparable or even better novel view reconstruction quality comparing with state-of-the-art baselines while increasing rendering speed by over 400x. Our model is trained in a category-agnostic manner and does not require scene-specific optimization. Therefore, it is able to generalize novel view synthesis to object categories not seen during training. In addition, we show that with our simple formulation, we can use view synthesis as a self-supervision signal for efficient learning of 3D geometry without explicit 3D supervision.

Introduction

Novel View Synthesis (NVS) is the task of generating new images of a scene from a different viewpoint given only a few source observations. Recently, the field has been dominated by implicit representations, such as Neural Radiance Fields (NeRF), which represent 3D scenes as continuous functions. While these methods produce high-resolution results, they suffer from significant computational costs and often require hours of scene-specific optimization.

This paper challenges the trend toward implicit functions, arguing that explicit volumetric representations (voxels) are the key to making NVS practical for real-time applications like AR/VR or robotics. By explicitly encoding source images into a 3D voxel grid, the proposed model enables real-time rendering and generalizes to unseen object categories without requiring per-scene optimization.

Methodology

The proposed approach utilizes a "simple yet effective" pipeline consisting of three main stages:

  1. 2D Encoding & Inverse Projection: A 2D U-Net extracts feature maps from the source image. The model then uses inverse projection to back-project these features into a 3D latent volume based on camera intrinsic parameters, preserving spatial relationships.
  2. 3D Volumetric Processing: A 3D U-Net processes this latent volume to predict a final RGBα volume (a grid where each cell contains color and opacity). If multiple source views are available, their volumes are aligned and averaged.
  3. Amortized Rendering: Because the 3D geometry is explicitly calculated in the volume, rendering a new view does not require re-running a heavy neural network. The model simply warps the volume to the target pose and uses a differentiable volume rendering equation to produce the image.
The model learns an explicit volumetric representation from source views, enabling fast amortized rendering.

Experimental Results

The model was evaluated using ShapeNet datasets, including category-specific benchmarks (Chairs and Cars) and multi-category sets to test category-agnostic performance. Key results include:

  • Massive Speedup: The model achieves 245 FPS for rendering, compared to 0.5 FPS for pixelNeRF, representing a 490x improvement in throughput.
  • Competitive Quality: Despite the simplicity of voxels, the model matches or exceeds the visual quality (PSNR, SSIM, LPIPS) of state-of-the-art baselines across various ShapeNet benchmarks.
  • Unsupervised 3D Reconstruction: The model learns to create accurate 3D occupancy grids using only 2D image losses, outperforming existing unsupervised 3D reconstruction methods like PlatonicGAN by a significant margin (0.58 vs 0.20 mIoU).
A target view synthesized by the model, demonstrating high fidelity to the scene geometry and successful novel view generation.

Conclusion and Future Work

This work demonstrates that explicit volumetric representations can provide a highly efficient alternative to implicit methods for novel view synthesis. By predicting the entire 3D structure once in a single forward pass, the rendering cost is amortized, allowing for real-time performance.

However, voxel grids scale cubically, which can limit the sharpness of fine details. Future research may explore more efficient 3D structures like octrees or point clouds to increase resolution, as well as light-weight modules to handle view-dependent lighting effects like specularities.

Summary updated

Abstract

We study the problem of novel view synthesis from sparse source observations of a scene comprised of 3D objects. We propose a simple yet effective approach that is neither continuous nor implicit, challenging recent trends on view synthesis. Our approach explicitly encodes observations into a volumetric representation that enables amortized rendering. We demonstrate that although continuous radiance field representations have gained a lot of attention due to their expressive power, our simple approach obtains comparable or even better novel view reconstruction quality comparing with state-of-the-art baselines [49] while increasing rendering speed by over 400x. Our model is trained in a category-agnostic manner and does not require scene-specific optimization. Therefore, it is able to generalize novel view synthesis to object categories not seen during training. In addition, we show that with our simple formulation, we can use view synthesis as a self-supervision signal for efficient learning of 3D geometry without explicit 3D supervision.
BibTeX
@inproceedings{guo2022fast,
  author = {Pengsheng Guo and Miguel Ángel Bautista and Alex Colburn and Liang Yang and Daniel Ulbricht and Joshua M. Susskind and Qi Shan},
  title = {{Fast and Explicit Neural View Synthesis}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=hztnT3uEhpc},
}
Figure