Neural Pixel Composition for 3D-4D View Synthesis from Multi-Views

Aayush Bansal, Michael ZollhöferCVPR 2023 · OpenReview

Abstract

We present Neural Pixel Composition (NPC), a novel approach for continuous 3D-4D view synthesis given a discrete set of multi-view observations. Unlike state-of-the-art coordinate-based networks that require dense supervision and significant compute, NPC operates reliably on sparse, wide-baseline data. By representing pixels as rich descriptors accumulated from multiple views and employing a lightweight Multi-Layer Perceptron (MLP) for composition, NPC achieves high-resolution (12MP) rendering with training times reduced by orders of magnitude compared to traditional Neural Radiance Fields (NeRF).

Introduction

Generating novel views from a set of captured images is a fundamental challenge in computer vision. While Neural Radiance Fields (NeRF) have set a high bar for quality, they suffer from high computational costs, often requiring hours or days of training. Furthermore, NeRF typically necessitates dense camera coverage and struggles with high-resolution or unbounded scenes due to memory constraints.

Neural Pixel Composition (NPC) addresses these limitations by shifting the paradigm from volumetric integration to neural composition. Instead of learning a scene's volume from scratch, NPC leverages existing multi-view information to synthesize new perspectives efficiently.

Methodology: The NPC Framework

The core of NPC is the Rich Pixel Descriptor. For any given pixel in a target view, the system accumulates color, depth, and uncertainty information from available source views using off-the-shelf disparity estimation. This data forms a descriptor that serves as the input to a compact, 5-layer MLP.

NPC demonstrates high-quality synthesis across various datasets and camera densities, from 30 views down to as few as 9.

Unlike NeRF's ray-marching, which samples hundreds of points along a line of sight, NPC performs a single forward pass per pixel. The MLP learns to output blending weights and color correction terms, effectively "composing" the final pixel color from the multi-view descriptors. This approach is highly efficient, requiring only 1GB of GPU memory and allowing for training on 12MP content in minutes.

Key Insights and Generalization

A significant advantage of NPC is its use of geometry as a prior. By utilizing explicit depth estimates, the model focuses on resolving conflicts between views rather than discovering 3D structure. The inclusion of uncertainty values makes the system robust to noisy depth maps.

NPC's ability to handle complex facial expressions and generalize to unseen data.

Furthermore, NPC exhibits remarkable generalization. A model trained on a single time-instant of a face can generalize to new expressions and even different individuals without retraining, highlighting the robustness of the learned composition weights.

Experimental Results

NPC was evaluated on diverse datasets including Open4D, LLFF, and the Shiny Dataset. Key findings include:

  • Efficiency: NPC achieves results in 10 minutes that would take NeRF 64 hours, a 200–400x speedup.
  • Resolution: The system successfully handles 12MP content, far exceeding the 1-2MP typical of existing neural rendering methods.
  • Sparse Data: In wide-baseline settings where cameras are far apart, NPC maintains high fidelity where NeRF-based methods often produce blurry or degenerate outputs.

Conclusion

Neural Pixel Composition represents a significant step forward in making 3D and 4D view synthesis practical for high-resolution, real-world applications. By combining explicit geometric priors with efficient neural composition, NPC provides a scalable and robust alternative to volumetric rendering, particularly in data-constrained environments.

Summary updated

Abstract

We present Neural Pixel Composition (NPC), a novel approach for continuous 3D-4D view synthesis given a discrete set of multi-view observations as input. Existing state-of-the-art approaches require dense multi-view supervision and an extensive computational budget. The proposed formulation reliably operates on sparse and wide-baseline multi-view images/videos and can be trained efficiently within a few seconds to 10 minutes for hi-res (12MP) content. Crucial to our approach are two core novelties: 1) a representation of a pixel that contains color and depth information accumulated from multi-views for a particular location and time along a line of sight, and 2) a multi-layer perceptron (MLP) that enables the composition of this rich information provided for a pixel location to obtain the final color output. We experiment with a large variety of multi-view sequences, compare to existing approaches, and achieve better results in diverse and challenging settings.
BibTeX
@inproceedings{bansal2023neural,
  author = {Aayush Bansal and Michael Zollhöfer},
  title = {{Neural Pixel Composition for 3D-4D View Synthesis from Multi-Views}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=nveSnAxkD0},
}
Figure