Abstract
Synthesizing novel views from sparse inputs (2–8 images) is challenging due to the "hallucination gap" and "consistency gap." SparseGNV addresses this by integrating explicit 3D neural point clouds with a transformer-based generative framework. This approach allows the model to hallucinate unobserved regions while maintaining spatial consistency, outperforming existing NeRF-based and purely generative methods on datasets like ScanNet and Replica. The system operates in a feed-forward manner, enabling efficient inference without per-scene optimization.
Introduction
Novel View Synthesis (NVS) typically requires a dense set of images to reconstruct a 3D scene accurately. In practical indoor scenarios, however, capturing every angle is often impossible, leading to sparse input views. Existing solutions face a trade-off: Neural Radiance Fields (NeRFs) provide high consistency but fail to "hallucinate" missing regions, resulting in black voids or blur. Conversely, purely generative models can create realistic details but often struggle with spatial consistency, leading to artifacts as the camera moves. SparseGNV bridges this gap by marrying explicit 3D structures with generative transformers.
Methodology
The SparseGNV framework consists of a three-module pipeline designed to generate consistent, photorealistic images from sparse inputs.
- Neural Geometry Module: Utilizing Multi-View Stereo (MVSNet), this module estimates depth and constructs a 3D neural point cloud. Each point is associated with an embedding vector, providing a geometric skeleton that guides the target view generation even when the reconstruction is incomplete.
- View Generator Module: This module acts as the core reasoning engine. A Transformer Encoder maps the scene context (observed images and rendered point cloud previews) into a shared latent space. A Transformer Decoder then autoregressively predicts the target image as a sequence of discrete Vector Quantized (VQ) tokens.
- Image Converter Module: Based on the VQ-GAN architecture, this module reconstructs the predicted tokens into a high-resolution, photorealistic 2D image.
Experiments and Results
The model was primarily trained and evaluated on ScanNet, a large-scale dataset of real-world indoor scans, and tested for generalization on the high-quality synthetic Replica dataset.
Quantitative Performance
On the ScanNet test set with only 2 input views, SparseGNV achieved a PSNR of 15.248, significantly outperforming ViewFormer (14.365) and DDP-NeRF (14.281). It also demonstrated superior performance in structural similarity (SSIM) and perceptual quality (LPIPS) metrics.
Qualitative Analysis
SparseGNV excels at "hallucinating" realistic furniture and walls in areas not seen by the input cameras. While baseline methods like IBRNet often produce blurry patches or voids, SparseGNV generates stable, detailed surfaces.
Efficiency
Unlike standard NeRFs that require per-scene optimization, SparseGNV is a feed-forward model. It can process a new, unseen room in approximately 0.83 seconds for a batch of 24 images on an NVIDIA V100 GPU, making it suitable for interactive use.
Conclusion
SparseGNV demonstrates that combining explicit 3D geometry with generative transformers is a powerful strategy for sparse-view synthesis. By using a neural point cloud as a "guide rail," the model ensures that its hallucinations remain grounded in 3D space. Future work may address minor instabilities in VQ-token generation and the system's dependency on accurate initial pose and depth estimates.
Summary updated