Abstract
We present a method for synthesizing novel views from a single 360-degree image based on the neural radiance field (NeRF). Prior studies rely on the neighborhood interpolation capability of multi-layer perceptrons to complete missing regions caused by occlusion and zooming, and this leads to artifacts. In the proposed method, the input image is reprojected to 360-degree images at other camera positions, the missing regions of the reprojected images are completed by a self-supervised trained generative model, and the completed images are utilized to train the NeRF. Because multiple completed images contain inconsistencies in 3D, we introduce a method to train NeRF while dynamically selecting a sparse set of completed images, to reduce the discrimination error of the synthesized views with real images. Experiments indicate that the proposed method can synthesize plausible novel views while preserving the features of the scene for both artificial and real-world data.
Introduction
Novel View Synthesis (NVS) typically requires a dense set of images to reconstruct a photorealistic 3D scene. However, capturing environments with a single 360-degree RGB-D image is highly efficient but introduces significant challenges. Moving the virtual camera away from the capture point reveals "holes" in the scene caused by occlusions and resolution changes. Traditional methods like OmniNeRF rely on the internal interpolation of Neural Radiance Fields (NeRF) to fill these gaps, which often results in blurry artifacts or structural "collapse" in unobserved regions.
Methodology
The proposed method addresses these issues by integrating self-supervised omnidirectional image completion into the NeRF training pipeline. The process consists of four main stages:
- Reprojection: The input 360-degree RGB-D image is reprojected to multiple virtual camera positions, creating masked images where occluded regions are represented as missing data.
- Omnidirectional Completion: A specialized completion network (based on OmniDreamer) fills these missing regions with semantically plausible textures.
- Subset Selection: Since independently completed images may lack 3D consistency, training on all of them can cause blurring. The authors solve this by selecting a subset of images that maximize scene coverage while minimizing overlap in hallucinated regions. This is formulated as a Maximum Weight Independent Set Problem (MWISP) and solved via Simulated Annealing.
- NeRF Training: The optimized subset of completed images is used to train a high-speed NeRF model (Instant-NGP).
Datasets
The method was evaluated on two major indoor benchmarks:
- Structured3D: A large-scale synthetic dataset of 3,500 house designs with photorealistic panoramic renderings.
- Matterport3D: A real-world dataset of 10,800 panoramas, featuring sensor noise and missing polar data, providing a rigorous test for robustness.
Results and Discussion
The proposed approach consistently outperforms OmniNeRF. Qualitatively, it produces sharper textures and more stable geometry in occlusion zones, avoiding the "melting" effect common in previous methods. Quantitatively, on the Structured3D dataset, it achieved a lower Negative Log Likelihood (NLL) of 2.295 compared to the baseline's 2.316. Ablation studies confirmed that the subset selection strategy is critical; without it, the conflicting "hallucinations" from different views lead to increased blurring and worse perceptual scores (LPIPS).
Conclusion
While the method significantly improves NVS from single panoramas, it remains dependent on the quality of the initial depth map and the generative capacity of the completion network. Future work may involve using more advanced generative models, such as 360-degree diffusion models, to further enhance the realism of hallucinated regions and finding ways to enforce 3D consistency during the completion phase itself.
Summary updated