Multiplane NeRF-Supervised Disentanglement of Depth and Camera Pose from Videos

Yang Fu, Ishan Misra, Xiaolong Wang

Abstract

MonoNeRF is a novel autoencoder-based framework that learns generalizable Neural Radiance Fields (NeRFs) directly from monocular videos without requiring ground-truth camera poses or depth annotations. By introducing a specialized auto-scale calibration technique, it successfully disentangles camera motion from scene geometry, enabling high-quality depth estimation, pose tracking, and novel view synthesis across diverse indoor scenes.

Introduction

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, but they typically come with two major limitations: the requirement for precise camera poses (usually calculated via time-consuming pre-processing like Structure-from-Motion) and a lack of generalization across different scenes. MonoNeRF addresses these challenges by learning a representation that generalizes across diverse environments using only raw, unposed monocular video as input.

Methodology

The MonoNeRF architecture is an autoencoder-based system designed to reconstruct video frames through a 3D-aware bottleneck.

Encoders and Decoder

The framework utilizes two specialized encoders: one for estimating monocular depth per frame and another for predicting relative camera poses (rotation and translation) between consecutive frames. Instead of a standard coordinate-based MLP, the decoder employs a Multiplane NeRF representation. This constructs a set of parallel feature planes from the depth encoder's output, allowing the model to handle occlusions and continuous depth more effectively than simple image warping.

Auto-Scale Calibration

A core challenge in self-supervised 3D learning is scale ambiguity—where different combinations of camera movement and scene depth can produce identical image sequences. MonoNeRF introduces a novel auto-scale calibration loss. This constraint forces the estimated monocular depth, the rendered NeRF depth, and the predicted camera trajectory to align on a consistent physical scale, which is critical for accurate 3D reasoning.

Experimental Results

The model was evaluated on major indoor datasets including ScanNet, NYU Depth V2, and RealEstate10K.

Depth Estimation and Pose Tracking

MonoNeRF achieves state-of-the-art performance in self-supervised depth estimation. On the NYU Depth V2 dataset, it outperformed previous methods like Monodepth2 and achieved results comparable to some supervised models.

Qualitative results of depth estimation across diverse indoor scenes, showing input frames and corresponding predicted depth maps.

In terms of camera trajectory, the model showed significant gains. On the RealEstate10K dataset, MonoNeRF reduced the Root Mean Square Error (RMSE) of camera trajectories from 0.055 to 0.011—an approximately 80% improvement over previous self-supervised baselines.

Comparison of the estimated camera trajectory (aligned) against the reference ground truth, demonstrating high precision in pose tracking.

Novel View Synthesis

By understanding the underlying 3D structure rather than just pixel movement, MonoNeRF produces sharper and more geometrically consistent novel views compared to traditional appearance flow methods.

Discussion and Limitations

While MonoNeRF successfully removes the requirement for Structure-from-Motion (SfM) pre-processing, it currently assumes static scenes. Moving objects like people or pets would likely degrade performance. Additionally, the Multiplane NeRF representation, while efficient for generalization, requires significant memory for high-resolution feature planes and is currently optimized for indoor environments with finite depth ranges.

Conclusion

MonoNeRF represents a significant step toward "unconstrained" NeRF training. By disentangling pose and depth through auto-scale calibration and a multiplane decoder, it enables high-fidelity 3D reconstruction and novel view synthesis from simple monocular video clips without the need for external pose or depth supervision.

Summary updated

Abstract

We propose to perform self-supervised disentanglement of depth and camera pose from large-scale videos. We introduce an Autoencoder-based method to reconstruct the input video frames for training, without using any ground-truth annotations of depth and camera. The encoders for our model will estimate the monocular depth and camera pose as the disentangled representations. The decoder will then construct a Multiplane NeRF representation based on the depth encoder feature, and perform rendering to reconstruct the input frames with the estimated camera. The disentanglement is learned with the reconstruction error, based on the assumption that the scene structure does not change in short periods of time in videos. Once the model is learned, it can be applied to multiple applications including depth estimation, camera pose estimation, and single image novel view synthesis. We show substantial improvements over previous self-supervised approaches on all tasks and even better results than counterparts trained with camera ground-truths in some applications. Our code will be made publicly available.
BibTeX
@misc{fu2022multiplane,
  author = {Yang Fu and Ishan Misra and Xiaolong Wang},
  title = {{Multiplane NeRF-Supervised Disentanglement of Depth and Camera Pose from Videos}},
  year = {2022},
  url = {https://openreview.net/forum?id=uefYIUDf95},
}
Figure