Point-DynRF: Point-based Dynamic Radiance Fields from a Monocular Video

Byeongjun Park, Changick Kim

Abstract

Dynamic radiance fields have emerged as a promising approach for generating novel views from a monocular video. However, previous methods enforce the geometric consistency to dynamic radiance fields only between adjacent input frames, making it difficult to represent the global scene geometry and degenerates at the viewpoint that is spatio-temporally distant from the input camera trajectory. To solve this problem, we introduce point-based dynamic radiance fields (Point-DynRF), a novel framework where the global geometric information and the volume rendering process are trained by neural point clouds and dynamic radiance fields, respectively. Specifically, we reconstruct neural point clouds directly from geometric proxies and optimize both radiance fields and the geometric proxies using our proposed losses, allowing them to complement each other. We validate the effectiveness of our method with experiments on the NVIDIA Dynamic Scenes Dataset and several causally captured monocular video clips.

Introduction

Creating novel view synthesis—the ability to navigate a virtual camera through a scene—from a single moving video is exceptionally challenging when the scene is dynamic. While standard Neural Radiance Fields (NeRF) excel at static environments, dynamic scenes introduce significant ambiguity: determining whether a change in pixels is caused by camera motion or object movement.

Existing methods typically rely on "local consistency," using adjacent frames to estimate motion via optical flow. However, when a camera follows a wide trajectory or a subject moves across a large area, these local methods often lose track of global structure. This leads to "ghosting" or duplicated objects in rendered views because the model cannot reconcile geometry over long time intervals.

Methodology: Neural Point Clouds

Point-DynRF addresses these challenges by shifting the scene representation from a continuous volume to a Neural Point Cloud. This approach allows the model to maintain a "global memory" of the scene; even if a background element is occluded for many frames, its 3D points persist in the cloud, ensuring accurate rendering upon reappearance.

Visualization of generated dynamic point clouds. The top row shows the aggregated point cloud, while the bottom rows show point clouds at specific timestamps.

Core Components

  1. Point Cloud Reconstruction: The system utilizes off-the-shelf depth estimation (DPT) and foreground masks to initialize 3D points. Each point is assigned a position, timestamp, feature vector, and a rigidness value ($\gamma$), which distinguishes between static background and moving objects.
  2. Dynamic Ray Marching: To prevent dynamic objects from "bleeding" into incorrect time steps, the model employs a novel sampling scheme. It selects only highly rigid points (static background) or points matching the current rendering timestamp.
  3. Joint Optimization: Rather than treating initial depth as ground truth, Point-DynRF treats it as a proxy. The system jointly optimizes radiance fields and geometric proxies (adjusting point positions, scale, and shift) using specialized losses, including a "missing ray" loss to handle sparse point cloud regions.

Results and Key Insights

Point-DynRF was evaluated on the NVIDIA Dynamic Scenes Dataset and various real-world monocular clips. The method achieved state-of-the-art SSIM scores, indicating superior structural preservation compared to baselines like DVS and RoDynRF.

Overview of Point-DynRF showing input monocular video, generated dynamic point clouds, and a comparison of fixed-viewpoint rendering results against DVS.

A primary advantage of Point-DynRF is its robustness to wide camera trajectories. While flow-based methods like DVS often produce infinite duplicates of moving subjects when the camera moves far from the original path, Point-DynRF’s temporal sampling ensures dynamic objects are only rendered at their specific spatio-temporal coordinates. Furthermore, the model is relatively efficient, requiring approximately 20 GPU hours for training, significantly less than the 36-48 hours required by competing frameworks.

Limitations and Future Work

The framework remains dependent on accurate camera poses from tools like COLMAP; inaccuracies can lead to blurry or misaligned neural point clouds. Additionally, the model is sensitive to initial depth map quality. Future research will focus on optimizing the neighbor search required during rendering to move toward real-time performance and reducing sensitivity to initialization.

Summary updated

Abstract

Dynamic radiance fields have emerged as a promising approach for generating novel views from a monocular video. However, previous methods enforce the geometric consistency to dynamic radiance fields only between adjacent input frames, making it difficult to represent the global scene geometry and degenerates at the viewpoint that is spatio-temporally distant from the input camera trajectory. To solve this problem, we introduce point-based dynamic radiance fields (\textbf{Point-DynRF}), a novel framework where the global geometric information and the volume rendering process are trained by neural point clouds and dynamic radiance fields, respectively. Specifically, we reconstruct neural point clouds directly from geometric proxies and optimize both radiance fields and the geometric proxies using our proposed losses, allowing them to complement each other. We validate the effectiveness of our method with experiments on the NVIDIA Dynamic Scenes Dataset and several causally captured monocular video clips.
BibTeX
@inproceedings{park2024point,
  author = {Byeongjun Park and Changick Kim},
  title = {{Point-DynRF: Point-based Dynamic Radiance Fields from a Monocular Video}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=MN3QG2jeFo},
}
Figure