Abstract
Dynamic radiance fields have emerged as a promising approach for generating novel views from a monocular video. However, previous methods enforce the geometric consistency to dynamic radiance fields only between adjacent input frames, making it difficult to represent the global scene geometry and degenerates at the viewpoint that is spatio-temporally distant from the input camera trajectory. To solve this problem, we introduce point-based dynamic radiance fields (Point-DynRF), a novel framework where the global geometric information and the volume rendering process are trained by neural point clouds and dynamic radiance fields, respectively. Specifically, we reconstruct neural point clouds directly from geometric proxies and optimize both radiance fields and the geometric proxies using our proposed losses, allowing them to complement each other. We validate the effectiveness of our method with experiments on the NVIDIA Dynamic Scenes Dataset and several causally captured monocular video clips.
Introduction
Synthesizing new viewpoints from a single (monocular) video is a long-standing goal in computer vision. However, it remains exceptionally difficult because the camera captures the scene from only one perspective at any given time, making it hard to distinguish between static backgrounds and moving objects. Existing techniques, such as deformable NeRFs or flow-based models, often struggle when the camera moves significantly or when rendering views far from the original path. These models frequently produce "ghosting" artifacts—where moving objects appear duplicated—or lose background detail as soon as the camera deviates from its original trajectory. This occurs because these models lack a consistent, global understanding of the 3D scene geometry over time.
Methodology
Point-DynRF addresses these challenges by shifting the scene representation from purely coordinate-based Multi-Layer Perceptrons (MLPs) to a neural point cloud. This approach "anchors" radiance to physical surfaces, providing a much stronger geometric prior than standard MLPs, which must learn the entire 3D structure from scratch.
Point Cloud Reconstruction and Representation
The system takes the input video and estimates initial depth and foreground masks using off-the-shelf tools. These are then back-projected into 3D space to create a cloud of points, each carrying a learnable feature vector. Unlike previous methods that treat initial depth maps as absolute truth, Point-DynRF treats them as "proxies" that are refined during training through specialized losses.
Dynamic Ray Marching and Aggregation
A core innovation is the dynamic ray marching strategy. Instead of sampling every point for every frame, the model selectively samples a subset of points at each time step $t$. This subset includes all static background points and only the dynamic points captured at that specific time $t$. This ensures that a moving object exists only at its correct physical location for that moment, mathematically preventing duplication. When rendering a ray, the model aggregates features from the $K=8$ nearest neural points to compute local volume density and color.
Experiments and Results
The authors evaluated Point-DynRF on the NVIDIA Dynamic Scenes Dataset and several causally captured handheld monocular videos. These real-world clips are particularly challenging due to wide-range camera trajectories that typically break existing models.
Performance and Robustness
Point-DynRF achieved superior image quality, specifically the highest SSIM (Structural Similarity Index) across all tested scenes in the NVIDIA dataset, indicating better preservation of textures and structures. In "Extremely Wide-Range" tests, Point-DynRF maintained a clear background and a single, sharp dynamic object, while competing methods like DVS produced multiple "ghost" copies of the subject.
Efficiency
The model is relatively fast to train, requiring only 20 GPU hours compared to 32–48 hours for some baselines. However, rendering is slightly slower (~11 seconds per image) due to the overhead of searching for neighboring points ($K$-nearest neighbor search).
Discussion and Conclusion
By using a point-based representation, Point-DynRF avoids wasting computation on empty space and produces sharper boundaries for moving objects. Tying dynamic points to specific timestamps effectively solves the "ghosting" problem. While the method relies on accurate camera poses (e.g., from COLMAP), it demonstrates significant robustness to wide camera motion. Future work may involve optimizing the neighbor search using data structures like octrees to improve rendering speed and extending the representation to handle more complex non-rigid deformations like splashing water.
Summary updated