Self-Calibrating 4D Novel View Synthesis from Monocular Videos Using Gaussian Splatting

Fang Li, Hao Zhang, Narendra Ahuja

Abstract

Gaussian Splatting (GS) has significantly elevated scene reconstruction efficiency and novel view synthesis (NVS) accuracy compared to Neural Radiance Fields (NeRF), particularly for dynamic scenes. However, current 4D NVS methods, whether based on GS or NeRF, primarily rely on camera parameters provided by COLMAP and even utilize sparse point clouds generated by COLMAP for initialization, which lack accuracy as well are time-consuming. This sometimes results in poor dynamic scene representation, especially in scenes with large object movements, or extreme camera conditions e.g. small translations combined with large rotations. Some studies simultaneously optimize the estimation of camera parameters and scenes, supervised by additional information like depth, optical flow, etc. obtained from off-the-shelf models. Using this unverified information as ground truth can reduce robustness and accuracy, which does frequently occur for long monocular videos (with e.g. > hundreds of frames). We propose a novel approach that learns a high-fidelity 4D GS scene representation with self-calibration of camera parameters. It includes the extraction of 2D point features that robustly represent 3D structure, and their use for subsequent joint optimization of camera parameters and 3D structure towards overall 4D scene optimization. We demonstrate the accuracy and time efficiency of our method through extensive quantitative and qualitative experimental results on several standard benchmarks. The results show significant improvements over state-of-the-art methods for 4D novel view synthesis. The source code will be released soon at https://github.com/fangli333/SC-4DGS.

Introduction

Gaussian Splatting (GS) has revolutionized scene reconstruction and novel view synthesis (NVS), particularly for dynamic scenes. However, existing 4D NVS methods often rely on external tools like COLMAP for camera parameters and sparse point cloud initialization. This dependency introduces significant bottlenecks: COLMAP struggles with dynamic objects, fails in "small translation, large rotation" scenarios common in handheld videos, and is computationally expensive. While some self-calibrating methods exist, they often depend on unverified priors like depth or optical flow from off-the-shelf models, which can compromise robustness in long monocular videos. SC-4DGS aims to bridge this gap by creating a robust, fast, and self-contained system for 4D NVS.

Methodology

We introduce SC-4DGS (Self-Calibrating 4D Gaussian Splatting), a framework for high-fidelity 4D GS scene representation with integrated camera self-calibration. The pipeline consists of three primary stages:

  1. Structural Points Extraction (SPE): To avoid noisy tracking in low-texture areas, SPE identifies representative points using Canny edge detection and gradient magnitudes. Motion masks are applied to exclude dynamic objects, ensuring the calibration is based on the static background.
  2. Joint Optimization: The system simultaneously optimizes camera parameters (represented by quaternions and translations) and the 3D positions of the structural points. This process is guided by a projection loss (minimizing the distance between projected 3D points and detected 2D features) and depth regularization.
  3. 4D Gaussian Splatting: Following calibration, the model learns a 4D representation using a Canonical Field for static elements and a Deformation Field for dynamic ones, rendered through efficient Differential Gaussian Rasterization.
The SC-4DGS pipeline overview, highlighting the Structural Points Extraction (SPE) and the joint optimization of camera parameters and 3D structure.

Experimental Results

The proposed method was evaluated on standard benchmarks including NeRF-DS, DAVIS, and Nvidia. A key finding is the system's robustness; on the DAVIS dataset, where COLMAP fails in over 80% of scenes due to extreme camera motion or dynamic complexity, SC-4DGS successfully reconstructs high-fidelity scenes.

In terms of efficiency, SC-4DGS significantly outperforms previous self-calibrating state-of-the-art models. For instance, on the NeRF-DS "bell" scene, our method requires only 5 hours for total optimization, compared to 30 hours for RoDynRF. Quantitatively, SC-4DGS achieves PSNR, SSIM, and LPIPS scores comparable to methods using ground-truth COLMAP cameras. For the "cup" scene in NeRF-DS, it reached a PSNR of 35.94, nearly matching the 36.42 of COLMAP-initialized models.

Qualitative comparison of novel view synthesis results. SC-4DGS produces sharper and more accurate reconstructions compared to baselines like RoDynRF and Deformable-3DGS.

Conclusion

SC-4DGS provides a robust and efficient solution for 4D NVS from monocular videos by eliminating the dependency on pre-computed camera parameters. By focusing on structural points for joint optimization, it handles challenging dynamic scenes that cause traditional SfM tools to fail. While the current model assumes a constant focal length and requires motion masks, it represents a significant step toward fully autonomous 4D scene reconstruction.

Summary updated

Abstract

Gaussian Splatting (GS) has significantly elevated scene reconstruction efficiency and novel view synthesis (NVS) accuracy compared to Neural Radiance Fields (NeRF), particularly for dynamic scenes. However, current 4D NVS methods, whether based on GS or NeRF, primarily rely on camera parameters provided by COLMAP and even utilize sparse point clouds generated by COLMAP for initialization, which lack accuracy as well are time-consuming. This sometimes results in poor dynamic scene representation, especially in scenes with large object movements, or extreme camera conditions e.g. small translations combined with large rotations. Some studies simultaneously optimize the estimation of camera parameters and scenes, supervised by additional information like depth, optical flow, etc. obtained from off-the-shelf models. Using this unverified information as ground truth can reduce robustness and accuracy, which does frequently occur for long monocular videos (with e.g. > hundreds of frames). We propose a novel approach that learns a high-fidelity 4D GS scene representation with self-calibration of camera parameters. It includes the extraction of 2D point features that robustly represent 3D structure, and their use for subsequent joint optimization of camera parameters and 3D structure towards overall 4D scene optimization. We demonstrate the accuracy and time efficiency of our method through extensive quantitative and qualitative experimental results on several standard benchmarks. The results show significant improvements over state-of-the-art methods for 4D novel view synthesis. The source code will be released soon at https://github.com/fangli333/SC-4DGS.
BibTeX
@misc{li2024self,
  author = {Fang Li and Hao Zhang and Narendra Ahuja},
  title = {{Self-Calibrating 4D Novel View Synthesis from Monocular Videos Using Gaussian Splatting}},
  year = {2024},
  eprint = {2406.01042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.01042},
}
Figure