X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360deg Insufficient RGB-D Views

Haoyi ZhuWACV 2023

TL;DR

This paper introduces X-NeRF, a novel approach for synthesizing novel views of 360° scenes using only a few sparse and non-overlapping RGB-D images. Unlike previous methods that require dense views and train per-scene models, X-NeRF learns a general scene completion model that can handle multiple scenes, significantly reducing data and computational requirements for practical applications.

What Problem Does This Paper Address?

Neural Radiance Fields (NeRFs) have revolutionized novel view synthesis, but their reliance on dense input views makes them impractical for many real-world scenarios where data acquisition is expensive or limited. Existing methods often train a separate model for each scene, hindering scalability. Furthermore, while RGB-D data is becoming more accessible, many NeRF variants primarily leverage RGB information. This paper tackles the challenging problem of synthesizing high-quality novel views from extremely sparse, 360° RGB-D inputs across multiple scenes, a setting that is particularly prone to overfitting and difficult for implicit scene representations.

Methods

X-NeRF proposes a fully explicit approach to scene completion. Instead of implicitly encoding scenes in neural network parameters, it learns a general scene-irrelevant completion relationship. The core idea is to transform sparse RGB-D inputs into colorful point clouds, which are then voxelized into Minkowski sparse tensors. A 3D sparse generative Convolutional Neural Network (CNN), structured as a UNet-like encoder-decoder, processes these tensors. The encoder extracts spatial features from existing coordinates, while the decoder uses generative transposed convolutions and pruning layers to up-sample and complete the explicit neural radiance field. This explicit representation allows for fast volumetric rendering without requiring network inference during runtime. To combat overfitting from sparse inputs, X-NeRF employs a combination of rendering loss, perceptual loss, and view augmentation through random rotation of point clouds.

Datasets

The authors introduce a new dataset for this challenging task. It consists of 7 RGB-D cameras capturing 6 "seen" scenes and 4 "novel" scenes, where a robot arm performs various tasks. The views are intentionally sparse, with less than 10-20% overlap. For evaluation, one view per scene is reserved for testing, while the other six are used for training.

Key Insights and Ideas

  • Explicit Scene Completion: The shift from implicit scene encoding to an explicit completion task is crucial for handling sparse and multi-scene data. This allows the model to learn a generalizable "completion" strategy rather than scene-specific parameters.
  • 3D Sparse Convolutions: Leveraging Minkowski Engine for sparse tensor operations is key to efficiently processing the sparse point cloud data and enabling the 3D CNN architecture.
  • Multi-Stage Decoder with Pruning: The multi-stage decoder with progressive up-sampling and pruning helps stabilize training and manage computational resources by removing redundant points.
  • View Augmentation for Sparsity: Random rotation of input point clouds acts as a powerful augmentation technique to simulate unseen view directions, significantly improving robustness in sparse settings.
  • Multi-Scene Generalization: The explicit, scene-irrelevant completion approach naturally lends itself to multi-scene training, outperforming methods that are designed for single scenes or require reference views.

Key Results

X-NeRF demonstrates significant improvements over state-of-the-art implicit methods like DS-NeRF and DVGO on single-scene experiments, particularly in depth error. In multi-scene comparisons, X-NeRF outperforms pixelNeRF and IBRNet, achieving better novel view synthesis quality. Notably, the multi-scene trained X-NeRF even surpasses its single-scene counterpart, highlighting its strong generalization capabilities. The paper also shows that X-NeRF can effectively complete missing depth information and correct errors from low-cost depth cameras. Inference is efficient due to the explicit representation, requiring no neural network computations at runtime.

Limitations and Future Work

The authors acknowledge that while pruning is optional and can reduce memory, it might slightly affect execution speed. The paper focuses on the challenging setting of insufficient views and multi-scene synthesis. Future work could explore further optimizations for inference speed and investigate the application of X-NeRF to more complex dynamic scenes or scenarios with even greater view sparsity. The effectiveness of the proposed explicit completion approach suggests potential for broader applications in robotics and augmented reality where data acquisition is constrained.

Summary updated

BibTeX
@inproceedings{zhu2023x,
  author = {Haoyi Zhu},
  title = {{X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360deg Insufficient RGB-D Views}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure