Abstract
We present Neural Pixel Composition (NPC), a novel approach for continuous 3D-4D view synthesis given a discrete set of multi-view observations. Unlike state-of-the-art coordinate-based networks that require dense supervision and significant compute, NPC operates reliably on sparse, wide-baseline data. By representing pixels as rich descriptors accumulated from multiple views and employing a lightweight Multi-Layer Perceptron (MLP) for composition, NPC achieves high-resolution (12MP) rendering with training times reduced by orders of magnitude compared to traditional Neural Radiance Fields (NeRF).
Introduction
Generating novel views from a set of captured images is a fundamental challenge in computer vision. While Neural Radiance Fields (NeRF) have set a high bar for quality, they suffer from high computational costs, often requiring hours or days of training. Furthermore, NeRF typically necessitates dense camera coverage and struggles with high-resolution or unbounded scenes due to memory constraints.
Neural Pixel Composition (NPC) addresses these limitations by shifting the paradigm from volumetric integration to neural composition. Instead of learning a scene's volume from scratch, NPC leverages existing multi-view information to synthesize new perspectives efficiently.
Methodology: The NPC Framework
The core of NPC is the Rich Pixel Descriptor. For any given pixel in a target view, the system accumulates color, depth, and uncertainty information from available source views using off-the-shelf disparity estimation. This data forms a descriptor that serves as the input to a compact, 5-layer MLP.
Unlike NeRF's ray-marching, which samples hundreds of points along a line of sight, NPC performs a single forward pass per pixel. The MLP learns to output blending weights and color correction terms, effectively "composing" the final pixel color from the multi-view descriptors. This approach is highly efficient, requiring only 1GB of GPU memory and allowing for training on 12MP content in minutes.
Key Insights and Generalization
A significant advantage of NPC is its use of geometry as a prior. By utilizing explicit depth estimates, the model focuses on resolving conflicts between views rather than discovering 3D structure. The inclusion of uncertainty values makes the system robust to noisy depth maps.
Furthermore, NPC exhibits remarkable generalization. A model trained on a single time-instant of a face can generalize to new expressions and even different individuals without retraining, highlighting the robustness of the learned composition weights.
Experimental Results
NPC was evaluated on diverse datasets including Open4D, LLFF, and the Shiny Dataset. Key findings include:
- Efficiency: NPC achieves results in 10 minutes that would take NeRF 64 hours, a 200–400x speedup.
- Resolution: The system successfully handles 12MP content, far exceeding the 1-2MP typical of existing neural rendering methods.
- Sparse Data: In wide-baseline settings where cameras are far apart, NPC maintains high fidelity where NeRF-based methods often produce blurry or degenerate outputs.
Conclusion
Neural Pixel Composition represents a significant step forward in making 3D and 4D view synthesis practical for high-resolution, real-world applications. By combining explicit geometric priors with efficient neural composition, NPC provides a scalable and robust alternative to volumetric rendering, particularly in data-constrained environments.
Summary updated