Abstract
MonoNeRF is a novel autoencoder-based framework that learns generalizable Neural Radiance Fields (NeRFs) directly from monocular videos without requiring ground-truth camera poses or depth annotations. By introducing a specialized auto-scale calibration technique, it successfully disentangles camera motion from scene geometry, enabling high-quality depth estimation, pose tracking, and novel view synthesis across diverse indoor scenes.
Introduction
Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, but they typically come with two major limitations: the requirement for precise camera poses (usually calculated via time-consuming pre-processing like Structure-from-Motion) and a lack of generalization across different scenes. MonoNeRF addresses these challenges by learning a representation that generalizes across diverse environments using only raw, unposed monocular video as input.
Methodology
The MonoNeRF architecture is an autoencoder-based system designed to reconstruct video frames through a 3D-aware bottleneck.
Encoders and Decoder
The framework utilizes two specialized encoders: one for estimating monocular depth per frame and another for predicting relative camera poses (rotation and translation) between consecutive frames. Instead of a standard coordinate-based MLP, the decoder employs a Multiplane NeRF representation. This constructs a set of parallel feature planes from the depth encoder's output, allowing the model to handle occlusions and continuous depth more effectively than simple image warping.
Auto-Scale Calibration
A core challenge in self-supervised 3D learning is scale ambiguity—where different combinations of camera movement and scene depth can produce identical image sequences. MonoNeRF introduces a novel auto-scale calibration loss. This constraint forces the estimated monocular depth, the rendered NeRF depth, and the predicted camera trajectory to align on a consistent physical scale, which is critical for accurate 3D reasoning.
Experimental Results
The model was evaluated on major indoor datasets including ScanNet, NYU Depth V2, and RealEstate10K.
Depth Estimation and Pose Tracking
MonoNeRF achieves state-of-the-art performance in self-supervised depth estimation. On the NYU Depth V2 dataset, it outperformed previous methods like Monodepth2 and achieved results comparable to some supervised models.
In terms of camera trajectory, the model showed significant gains. On the RealEstate10K dataset, MonoNeRF reduced the Root Mean Square Error (RMSE) of camera trajectories from 0.055 to 0.011—an approximately 80% improvement over previous self-supervised baselines.
Novel View Synthesis
By understanding the underlying 3D structure rather than just pixel movement, MonoNeRF produces sharper and more geometrically consistent novel views compared to traditional appearance flow methods.
Discussion and Limitations
While MonoNeRF successfully removes the requirement for Structure-from-Motion (SfM) pre-processing, it currently assumes static scenes. Moving objects like people or pets would likely degrade performance. Additionally, the Multiplane NeRF representation, while efficient for generalization, requires significant memory for high-resolution feature planes and is currently optimized for indoor environments with finite depth ranges.
Conclusion
MonoNeRF represents a significant step toward "unconstrained" NeRF training. By disentangling pose and depth through auto-scale calibration and a multiplane decoder, it enables high-fidelity 3D reconstruction and novel view synthesis from simple monocular video clips without the need for external pose or depth supervision.
Summary updated