RefiNeRF: Modelling dynamic neural radiance fields with inconsistent or missing camera parameters

Shuja Khalid, Frank RudziczarXiv 2023 · DOI · arXiv

RefiNeRF is a modular framework designed to train Neural Radiance Fields (NeRFs) even when camera positions and lens properties are inaccurate or entirely missing. By optimizing camera parameters directly through a photometric loss and a specialized training scheduler, it enables high-quality 3D reconstruction of dynamic scenes in challenging environments where traditional tools like COLMAP fail.

The Challenge of Camera Calibration

To create a high-quality 3D scene, standard NeRF models require precise "camera poses"—the exact spatial coordinates and lens configuration for every frame. Typically, researchers rely on Structure-from-Motion (SfM) tools like COLMAP to calculate these parameters.

However, COLMAP is fragile. It often fails when images lack distinct textures (such as medical footage of internal organs), have low contrast, or involve complex moving objects. While previous works like BARF and NeRF– attempted to "learn" these parameters during training, they were largely limited to static scenes. RefiNeRF addresses the gap of performing joint camera calibration and scene reconstruction in dynamic environments where both the camera and the subjects are in motion.

Visualizing the impact of erroneous camera parameters on scene reconstruction. Erroneous extrinsics lead to ghosting/blurring, while erroneous intrinsics cause spatial shifts and cropping.

Methodology

RefiNeRF treats camera intrinsics (focal length, principal point) and extrinsics (rotation, translation) as learnable parameters that are optimized alongside the neural field itself.

Dual-Stream Architecture

The model separates the scene into static and dynamic components. One set of networks models the background, while another handles motion-centric pixels using a deformation network. This decoupling prevents moving objects from "polluting" the camera pose estimation, allowing the model to use the stable background to anchor the camera's position.

The RefiNeRF architecture, featuring separate static and dynamic streams, learnable camera models, and a renderer that facilitates gradient flow for parameter optimization.

Key Components

  • Photometric Optimization: Instead of relying on geometric keypoints, RefiNeRF uses the difference between the rendered image and the real image (photometric loss) to generate gradients. These gradients "push" the camera parameters toward their correct values.
  • Scheduled Learning: Updating all parameters simultaneously can lead to instability. The authors' scheduler first trains the static scene representation, then the dynamic components, and only begins refining camera parameters once the scene model has reached a baseline level of stability.
  • Multi-Resolution Grid Encoding: The model leverages efficient grid-based encoding to represent spatial and frequency data, allowing for faster convergence and higher detail.

Experimental Results

The authors evaluated RefiNeRF on two primary datasets: the NVIDIA Dynamic Scenes Dataset (featuring high-resolution motion like jumping and skating) and Cholec80 (laparoscopic surgery videos).

  • Superior Accuracy: On the NVIDIA dataset, RefiNeRF consistently outperformed state-of-the-art methods. In the "Jumping" scene, RefiNeRF achieved a PSNR of 13.224, compared to BARF’s 5.446, representing a massive leap in reconstruction quality for highly dynamic motion.
  • Surgical Success: The model successfully reconstructed scenes from the Cholec80 dataset where COLMAP failed entirely. It produced clear novel views of surgical environments where previous methods yielded only "failed" markers due to the smooth, textureless nature of internal tissues.
  • Robustness to Noise: The model can recover from significant initialization errors. Even if the initial camera guess is off by 50% in focal length or 30% in position, the photometric gradients are strong enough to pull the parameters back to the ground truth.

Limitations and Future Work

While RefiNeRF provides a robust solution for missing camera data, it inherits some classic NeRF drawbacks. The optimization process remains relatively slow due to dense 3D sampling requirements. Additionally, the training scheduler relies on manually defined thresholds that may require tuning for different scene types. Future research may involve integrating this refinement logic with faster backends like Instant-NGP to achieve near-real-time performance in applications like robotic surgery.

Summary updated

Abstract

Novel view synthesis (NVS) is a challenging task in computer vision that involves synthesizing new views of a scene from a limited set of input images. Neural Radiance Fields (NeRF) have emerged as a powerful approach to address this problem, but they require accurate knowledge of camera \textit{intrinsic} and \textit{extrinsic} parameters. Traditionally, structure-from-motion (SfM) and multi-view stereo (MVS) approaches have been used to extract camera parameters, but these methods can be unreliable and may fail in certain cases. In this paper, we propose a novel technique that leverages unposed images from dynamic datasets, such as the NVIDIA dynamic scenes dataset, to learn camera parameters directly from data. Our approach is highly extensible and can be integrated into existing NeRF architectures with minimal modifications. We demonstrate the effectiveness of our method on a variety of static and dynamic scenes and show that it outperforms traditional SfM and MVS approaches. The code for our method is publicly available at \href{https://github.com/redacted/refinerf}{https://github.com/redacted/refinerf}. Our approach offers a promising new direction for improving the accuracy and robustness of NVS using NeRF, and we anticipate that it will be a valuable tool for a wide range of applications in computer vision and graphics.
BibTeX
@misc{khalid2023refinerf,
  author = {Shuja Khalid and Frank Rudzicz},
  title = {{RefiNeRF: Modelling dynamic neural radiance fields with inconsistent or missing camera parameters}},
  year = {2023},
  eprint = {2303.08695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.08695},
}
Figure