Introduction
Recovering the underlying dynamic 3D scene representation from a monocular RGB video is a fundamental challenge in computer vision. Traditionally, this task has been treated as an optimization problem aimed at finding a single "most plausible" solution. However, monocular video is inherently ambiguous: a single 2D sequence can be explained by an infinite number of 3D configurations. For instance, a small object moving close to the lens can appear identical to a large object moving further away.
Existing methods, such as those based on Dynamic Neural Radiance Fields (NeRFs), typically address this ambiguity by imposing strong priors—such as monocular depth estimates or shape templates—to force the model toward a specific geometry. Unfortunately, these priors are often noisy, leading to distorted reconstructions. In this work, we introduce OSN (Object Scale Network), a framework that embraces this ambiguity. Instead of inferring a single configuration, OSN learns the entire range of plausible 3D scene representations consistent with the input video.
The OSN Framework
The core philosophy of OSN is to decouple object shape from its physical scale. The framework is built upon three primary innovations:
1. Object Scale-invariant Representations
Rather than modeling the entire scene as a single volume, OSN decomposes the scene into individual moving objects and a static background. Each entity is represented within its own "unit volume" using a high-performance backbone like TensoRF. By modeling objects in a normalized space, the network learns their intrinsic shape and appearance independently of their specific size or distance from the camera.
2. The Object Scale Network (OSN)
The namesake of our framework is a Multi-Layer Perceptron (MLP) designed to evaluate the physical consistency of different scale combinations. Given a set of scales for all objects in a scene, the OSN predicts a "validity score" (0 to 1). This allows the system to understand which relative sizes are physically possible given the visual evidence, such as occlusions and motion parallax. If a scale combination results in a "small" object appearing to occlude a "large" background object in a way that contradicts the video, the network marks that scale as invalid.
3. Joint Optimization and Rendering
To train these representations, we utilize a joint optimization module featuring two specialized rendering techniques:
- Scaled Composite Rendering: This method assembles individual objects at specific scales to reconstruct the original input frames.
- Soft Z-buffer Rendering: A differentiable approach to handling occlusions that allows the system to efficiently evaluate thousands of potential scale combinations during training.
The training process follows a two-stage pipeline: an initial "bootstrapping" phase to establish basic object shapes, followed by an "alternative optimization" where the scale network and the object representations are iteratively refined.
Experimental Validation
We evaluated OSN across a diverse range of synthetic and real-world datasets, including the Dynamic Indoor Scene Dataset, the Oxford Multimotion Dataset, and the KITTI autonomous driving benchmark.
Embracing Ambiguity: The "Gnome House" Experiment
To demonstrate OSN's unique capabilities, we designed a benchmark featuring 50 distinct ground-truth 3D configurations that all produce the same 2D video. While baseline models struggled to produce even one accurate version, OSN successfully sampled and matched all 50 valid configurations, proving its ability to capture the full distribution of plausible scenes.
Quantitative Performance
OSN consistently outperforms state-of-the-art baselines such as NSFF, DynNeRF, HexPlane, and Total-Recon:
- Novel View Synthesis: On the Indoor dataset, OSN achieved a PSNR of 25.98 dB, significantly higher than the next best competitor (24.69 dB).
- Geometry Accuracy: The improvement in geometry is even more pronounced. OSN reached a SSIMAE (Scale and Shift-Invariant Mean Absolute Error) of 0.094, a massive leap over the 0.137 achieved by Total-Recon.
Qualitatively, OSN avoids common reconstruction artifacts where objects appear to "sink" into the ground or "float" unnaturally, thanks to its superior understanding of relative object scales and occlusions.
Limitations and Future Directions
While OSN represents a significant step forward, it currently focuses on rigid moving objects and does not yet handle deformable entities like walking people. Additionally, the method relies on external preprocessing for segmentation (SAM) and initial poses (SfM). Future work will aim to extend this "infinite representation" logic to deformable scenes and integrate preprocessing into an end-to-end framework.
By shifting the focus from finding a single solution to capturing the entire space of possibilities, OSN provides a more robust and accurate foundation for dynamic 3D scene understanding from monocular video.
Summary updated