wildNeRF: Complete view synthesis of in-the-wild dynamic scenes captured using sparse monocular data

Shuja Khalid, Frank RudziczarXiv 2022 · DOI · arXiv

Introduction

Neural Radiance Fields (NeRF) have significantly advanced novel view synthesis for static environments. However, dynamic scenes—characterized by moving objects and non-rigid deformations—remain a significant challenge. Existing dynamic NeRF models often suffer from high computational costs, requiring days of training for seconds of output, and typically necessitate dense, multi-camera setups. Furthermore, "in-the-wild" monocular data, such as handheld camera footage or surgical videos, often lacks precise pose information, making traditional structure-from-motion (SfM) approaches unreliable.

Dynamic scene scenarios handled by wildNeRF: fixed world with moving camera (top) and moving world with fixed camera typical of surgical environments (bottom).

wildNeRF addresses these limitations through a self-supervised model capable of reconstructing complex dynamic scenes from sparse monocular video in minutes. By integrating flow-based and deformation-based modeling with an efficient pixel pruning technique, it achieves state-of-the-art performance while maintaining robustness in challenging real-world scenarios, such as laparoscopic surgery.

Methodology

The wildNeRF architecture is built upon a hybrid approach that decouples static and dynamic scene components.

Triple-Network Architecture

The system utilizes three lightweight, fully-connected networks to process different aspects of the scene:

  1. Background Network: Responsible for modeling static, rigid bodies within the environment.
  2. Deformation Network: Handles non-rigid dynamics, such as stretching or bending of objects.
  3. Scene-Flow Network: Predicts rigid-body motion between adjacent frames in both forward and backward temporal directions.
The wildNeRF Triple-Network Architecture, illustrating the flow from input coordinates and vectors through deformation, density, and color networks.

Efficiency and Encoding

To achieve high-speed training, wildNeRF employs Multi-Resolution Hash Encoding, which uses grid embeddings to accelerate the learning of high-frequency spatial details. A critical innovation is Pixel Pruning: the algorithm identifies and "turns off" spatial points that do not contribute to the final scene after an initial 5,000 iterations. This reduces matrix operations significantly, resulting in a 30x speedup compared to traditional methods.

Instead of complex learned blending, wildNeRF uses pre-calculated motion masks to determine which network processes a specific pixel, further reducing computational overhead and preventing "ghosting" artifacts.

Experimental Setup and Datasets

The model was evaluated across three diverse datasets to test its versatility:

  • NVIDIA Dynamic Scenes Dataset: A benchmark featuring 8 scenes captured via 12 synchronized cameras.
  • Cholec80: 80 full-length surgical videos of gallbladder removals, providing extreme "in-the-wild" conditions with specularities and non-rigid tissue.
  • SurgicalActions160: 160 clips of specific surgical tasks like suturing, representing a departure from idealized lab environments.

Results

wildNeRF demonstrates superior performance in both speed and accuracy:

  • Training Efficiency: The model trains in approximately 1.6 hours on a single RTX 3090, a massive improvement over the 48-hour training times required by original NeRF and its dynamic variants.
  • Quantitative Accuracy: On the NVIDIA Dynamic Scenes benchmark, wildNeRF achieved a 3% improvement in PSNR (Peak Signal-to-Noise Ratio) and a 1% improvement in SSIM (Structural Similarity Index) over previous state-of-the-art models.
  • Qualitative Robustness: The model successfully captures complex non-rigid motion and lighting effects (e.g., specularities on surgical tools) without the transparency or "disappearing" artifacts common in competing methods.

Limitations and Future Work

While highly effective, wildNeRF's performance is tied to the quality of its pre-calculated motion masks; poor masks can result in reconstruction artifacts or "holes." Additionally, aggressive pixel pruning can occasionally introduce blocky "grid errors" in the background. Resolving extremely fine or fast-moving details, such as needle tips in surgery, remains a challenge. Future research will focus on internalizing motion-mask generation and further optimizing grid-based encoders for real-time performance.

Summary updated

Abstract

We present a novel neural radiance model that is trainable in a self-supervised manner for novel-view synthesis of dynamic unstructured scenes. Our end-to-end trainable algorithm learns highly complex, real-world static scenes within seconds and dynamic scenes with both rigid and non-rigid motion within minutes. By differentiating between static and motion-centric pixels, we create high-quality representations from a sparse set of images. We perform extensive qualitative and quantitative evaluation on existing benchmarks and set the state-of-the-art on performance measures on the challenging NVIDIA Dynamic Scenes Dataset. Additionally, we evaluate our model performance on challenging real-world datasets such as Cholec80 and SurgicalActions160.
BibTeX
@misc{khalid2022wildnerf,
  author = {Shuja Khalid and Frank Rudzicz},
  title = {{wildNeRF: Complete view synthesis of in-the-wild dynamic scenes captured using sparse monocular data}},
  year = {2022},
  eprint = {2209.10399},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.10399},
}
Figure