MSI-NeRF: Linking Omni-Depth with View Synthesis through Multi-Sphere Image Aided Generalizable Neural Radiance Field

Dongyu Yan, Guanyu Huang, Fengyu Quan, Haoyao ChenWACV 2025

TL;DR

This paper introduces MSI-NeRF, a novel method that synthesizes a 3D scene representation from just four fisheye images, enabling both high-quality omnidirectional depth estimation and 6DoF novel view synthesis. By leveraging a "multi-sphere image" representation and a hybrid neural rendering approach, MSI-NeRF achieves generalizable performance across unseen scenes with significantly less input data than prior methods.

What Problem Does This Paper Address?

Traditional methods for creating panoramic views often rely on stitching multiple images, which can lead to stitching errors and loss of crucial depth information, limiting applications to 3DoF (rotation-only) rendering. While methods like NeRF have revolutionized novel view synthesis, they typically require scene-specific training or dense input views, making them impractical for real-world scenarios where capturing such data is challenging. Existing omnidirectional depth estimation methods also struggle with free-viewpoint rendering and can be sensitive to scene textures. This paper aims to bridge this gap by enabling accurate 3D scene reconstruction and 6DoF view synthesis from a sparse set of omnidirectional images.

Methods

MSI-NeRF combines several key innovations:

  • Multi-Sphere Image (MSI) Construction: The core idea is to create a richer intermediate representation. Features are extracted from the four input fisheye images using a 2D CNN. These features are then warped and aggregated into a "multi-sphere image" (MSI) across multiple inverse depth layers. This MSI acts as a cost volume, encoding geometric and appearance information from multiple viewpoints.
  • Hybrid Neural Rendering: A NeRF-like implicit neural representation (MLP) is used for scene rendering. Crucially, this MLP takes not only spatial coordinates and viewing directions but also interpolated features from the MSI as input. This allows the NeRF to leverage the pre-computed geometric and appearance priors from the MSI, enhancing its ability to generalize.
  • Multi-Task Supervision: To train the generalizable NeRF without requiring novel target views, the method uses a multi-task learning approach. It leverages existing omnidirectional depth estimation datasets for depth supervision (to learn occupancy) and uses the original input fisheye images for color supervision (to learn scene appearance). This allows for end-to-end training using readily available depth data.
  • Separated Geometry and Appearance Decoders: The MSI features are processed by two distinct decoders: one for geometry and one for appearance. This separation clarifies their roles and improves the overall learning process.

Datasets

  • Training: The method is trained on datasets from OmniMVS [38], specifically OmniThings and OmniHouse, which provide fisheye images and corresponding omnidirectional depth ground truth.
  • Evaluation:
    • The OmniHouse and OmniThings datasets are used for depth estimation evaluation.
  • A new benchmark, Replica360, is introduced for novel view synthesis evaluation. This dataset is constructed from the Replica [33] simulator and includes four fisheye source views and five omnidirectional target views.

Key Insights and Ideas

  • MSI as a Geometric Prior: The multi-sphere image acts as a powerful geometric prior, effectively aggregating information from sparse fisheye inputs into a structured representation that a NeRF can readily utilize. This is a key departure from methods that rely solely on raw image features or scene-specific optimization.
  • Leveraging Depth for Color: The insight that depth supervision can indirectly aid in learning scene appearance is crucial. By using depth to guide occupancy estimation, the network can then focus on learning color, even with limited direct color supervision from novel views.
  • Generalization from Sparse Inputs: The combination of MSI and a generalizable NeRF architecture allows for effective scene reconstruction and novel view synthesis from as few as four fisheye images, a significant reduction in input requirements.
  • Implicit Color Blending: Using projected pixel colors from the original fisheye images as an input to the NeRF acts as an implicit color blending mechanism, further improving the quality of synthesized views.

Key Results

  • Depth Estimation: MSI-NeRF achieves competitive or superior results compared to state-of-the-art methods like MatryODShka [1] and OmniMVS [38] on the OmniHouse and OmniThings datasets, producing finer-grained depth maps and reducing over-fitting on textured surfaces.
  • Novel View Synthesis: The method demonstrates excellent performance in novel view synthesis, outperforming MatryODShka [1] and NeRF-360-NGP [3, 29, 31] in terms of PSNR, SSIM, and LPIPS metrics. It generates high-quality, consistent renderings without blurring or ghosting artifacts, and shows strong generalization across datasets.
  • Efficiency: The method achieves efficient rendering with a single network inference.

Limitations and Future Work

  • Occluded Regions: The paper acknowledges that the method, being based on MVS principles, struggles with reconstructing occluded parts of the scene, leading to unpleasant results in these areas.
  • Calibration Accuracy: For real-world applications, accurate camera intrinsic and extrinsic parameters are crucial. Calibration errors can lead to blurring and artifacts in the generated images.

The authors suggest leveraging generative models to inpaint unseen parts of the scene as a future research direction.

Summary updated

BibTeX
@inproceedings{yan2025msi,
  author = {Dongyu Yan and Guanyu Huang and Fengyu Quan and Haoyao Chen},
  title = {{MSI-NeRF: Linking Omni-Depth with View Synthesis through Multi-Sphere Image Aided Generalizable Neural Radiance Field}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure