Introduction
Neural Radiance Fields (NeRF) have significantly advanced novel view synthesis for static environments. However, dynamic scenes—characterized by moving objects and non-rigid deformations—remain a significant challenge. Existing dynamic NeRF models often suffer from high computational costs, requiring days of training for seconds of output, and typically necessitate dense, multi-camera setups. Furthermore, "in-the-wild" monocular data, such as handheld camera footage or surgical videos, often lacks precise pose information, making traditional structure-from-motion (SfM) approaches unreliable.
wildNeRF addresses these limitations through a self-supervised model capable of reconstructing complex dynamic scenes from sparse monocular video in minutes. By integrating flow-based and deformation-based modeling with an efficient pixel pruning technique, it achieves state-of-the-art performance while maintaining robustness in challenging real-world scenarios, such as laparoscopic surgery.
Methodology
The wildNeRF architecture is built upon a hybrid approach that decouples static and dynamic scene components.
Triple-Network Architecture
The system utilizes three lightweight, fully-connected networks to process different aspects of the scene:
- Background Network: Responsible for modeling static, rigid bodies within the environment.
- Deformation Network: Handles non-rigid dynamics, such as stretching or bending of objects.
- Scene-Flow Network: Predicts rigid-body motion between adjacent frames in both forward and backward temporal directions.
Efficiency and Encoding
To achieve high-speed training, wildNeRF employs Multi-Resolution Hash Encoding, which uses grid embeddings to accelerate the learning of high-frequency spatial details. A critical innovation is Pixel Pruning: the algorithm identifies and "turns off" spatial points that do not contribute to the final scene after an initial 5,000 iterations. This reduces matrix operations significantly, resulting in a 30x speedup compared to traditional methods.
Instead of complex learned blending, wildNeRF uses pre-calculated motion masks to determine which network processes a specific pixel, further reducing computational overhead and preventing "ghosting" artifacts.
Experimental Setup and Datasets
The model was evaluated across three diverse datasets to test its versatility:
- NVIDIA Dynamic Scenes Dataset: A benchmark featuring 8 scenes captured via 12 synchronized cameras.
- Cholec80: 80 full-length surgical videos of gallbladder removals, providing extreme "in-the-wild" conditions with specularities and non-rigid tissue.
- SurgicalActions160: 160 clips of specific surgical tasks like suturing, representing a departure from idealized lab environments.
Results
wildNeRF demonstrates superior performance in both speed and accuracy:
- Training Efficiency: The model trains in approximately 1.6 hours on a single RTX 3090, a massive improvement over the 48-hour training times required by original NeRF and its dynamic variants.
- Quantitative Accuracy: On the NVIDIA Dynamic Scenes benchmark, wildNeRF achieved a 3% improvement in PSNR (Peak Signal-to-Noise Ratio) and a 1% improvement in SSIM (Structural Similarity Index) over previous state-of-the-art models.
- Qualitative Robustness: The model successfully captures complex non-rigid motion and lighting effects (e.g., specularities on surgical tools) without the transparency or "disappearing" artifacts common in competing methods.
Limitations and Future Work
While highly effective, wildNeRF's performance is tied to the quality of its pre-calculated motion masks; poor masks can result in reconstruction artifacts or "holes." Additionally, aggressive pixel pruning can occasionally introduce blocky "grid errors" in the background. Resolving extremely fine or fast-moving details, such as needle tips in surgery, remains a challenge. Future research will focus on internalizing motion-mask generation and further optimizing grid-based encoders for real-time performance.
Summary updated