OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos

Dongyoung Choi, Hyeonjoong Jang, Min H. KimCVPR 2024 · DOI · arXiv

Abstract

Neural Radiance Fields (NeRF) have revolutionized view synthesis, yet their application to casual 360° videos is often marred by dynamic objects and the persistent presence of the photographer. We present OmniLocalRF, a framework designed to reconstruct pristine, static 3D environments from long-trajectory omnidirectional videos. By leveraging local radiance fields, a self-supervised Bayesian motion masking module, and a bidirectional optimization strategy that utilizes distant frames for background inpainting, OmniLocalRF effectively eliminates ghosting artifacts. Our method operates without manual labels or pre-computed camera poses, outperforming existing state-of-the-art techniques in both visual quality and pose accuracy.

Introduction

The accessibility of consumer-grade omnidirectional cameras has simplified 360° content capture. However, transforming these videos into immersive 3D scenes via NeRF remains challenging due to dynamic elements. Standard NeRF models assume a static environment; moving pedestrians or the photographer manifest as "ghosting" artifacts—blurry, semi-transparent clouds that degrade visual fidelity.

OmniLocalRF addresses these issues specifically for long-trajectory 360° videos. It handles the inherent "blind spot" where the photographer is visible and eliminates the need for manual masking or external segmentation models, providing a fully automated pipeline for clean scene reconstruction.

Methodology

OmniLocalRF builds upon the Local Radiance Fields (LocalRF) approach, which partitions long videos into overlapping local blocks using the TensoRF backbone. This locality ensures scalability for videos of arbitrary length.

Bidirectional Optimization

Unlike perspective cameras, 360° cameras provide redundant spatial information. OmniLocalRF leverages "distant frames" for refinement. If an object occludes a background element in one frame, the system utilizes observations from frames where that element was visible (even if captured much earlier or later) to inpaint the current view accurately. This bidirectional constraint significantly improves the sharpness of reconstructed backgrounds.

Self-Supervised Motion Masking

To avoid reliance on pre-trained semantic segmentors, which may fail on shadows or non-standard objects, OmniLocalRF employs a Bayesian motion mask prediction module. By analyzing color and depth consistency across multi-resolution neural feature planes, the model learns to identify and mask dynamic pixels that do not conform to the underlying 3D geometry.

Joint Pose and Geometry Optimization

The framework does not require pre-computed camera trajectories from external tools like COLMAP. Instead, it jointly optimizes camera poses and scene geometry. By focusing optimization on regions identified as static by the motion mask, it achieves high tracking accuracy even in cluttered dynamic environments.

Experimental Results

The method was evaluated on synthetic and real-world 5.7K resolution 360° datasets. OmniLocalRF consistently outperformed state-of-the-art baselines including Mip-NeRF360 and EgoNeRF.

Comparison of OmniLocalRF against conventional neural rendering. The top row shows significant ghosting (red) and blur (yellow) from dynamic objects. The bottom row demonstrates OmniLocalRF's ability to remove these distractors and produce sharp, static background reconstructions.

Quantitatively, OmniLocalRF achieved a PSNR of 27.73 on real-world data, surpassing Mip-NeRF360 (26.88). Qualitatively, the method effectively removes "smudges" left by moving objects and the photographer, replacing them with sharp, inpainted textures. Synthetic tests further confirmed its robustness, showing significantly lower Absolute Trajectory Error (ATE) compared to standard Structure-from-Motion tools.

Discussion and Limitations

While OmniLocalRF excels at removing dynamic distractors, it is a reconstruction tool limited by the information present in the source video; it cannot recreate regions that are never visible. Future work may address equirectangular distortion through uniform spherical grids and integrate global bundle adjustment to mitigate potential drift in extremely long sequences. Despite these limitations, OmniLocalRF represents a significant step toward automated, high-quality 3D reconstruction from casual omnidirectional video.

Summary updated

BibTeX
@inproceedings{choi2024omnilocalrf,
  author = {Dongyoung Choi and Hyeonjoong Jang and Min H. Kim},
  title = {{OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00656},
}
Figure