OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos

Ziyang Song, Jinxi Li, Bo Yang

Introduction

Recovering the underlying dynamic 3D scene representation from a monocular RGB video is a fundamental challenge in computer vision. Traditionally, this task has been treated as an optimization problem aimed at finding a single "most plausible" solution. However, monocular video is inherently ambiguous: a single 2D sequence can be explained by an infinite number of 3D configurations. For instance, a small object moving close to the lens can appear identical to a large object moving further away.

Existing methods, such as those based on Dynamic Neural Radiance Fields (NeRFs), typically address this ambiguity by imposing strong priors—such as monocular depth estimates or shape templates—to force the model toward a specific geometry. Unfortunately, these priors are often noisy, leading to distorted reconstructions. In this work, we introduce OSN (Object Scale Network), a framework that embraces this ambiguity. Instead of inferring a single configuration, OSN learns the entire range of plausible 3D scene representations consistent with the input video.

The OSN Framework

The core philosophy of OSN is to decouple object shape from its physical scale. The framework is built upon three primary innovations:

1. Object Scale-invariant Representations

Rather than modeling the entire scene as a single volume, OSN decomposes the scene into individual moving objects and a static background. Each entity is represented within its own "unit volume" using a high-performance backbone like TensoRF. By modeling objects in a normalized space, the network learns their intrinsic shape and appearance independently of their specific size or distance from the camera.

2. The Object Scale Network (OSN)

The namesake of our framework is a Multi-Layer Perceptron (MLP) designed to evaluate the physical consistency of different scale combinations. Given a set of scales for all objects in a scene, the OSN predicts a "validity score" (0 to 1). This allows the system to understand which relative sizes are physically possible given the visual evidence, such as occlusions and motion parallax. If a scale combination results in a "small" object appearing to occlude a "large" background object in a way that contradicts the video, the network marks that scale as invalid.

3. Joint Optimization and Rendering

To train these representations, we utilize a joint optimization module featuring two specialized rendering techniques:

  • Scaled Composite Rendering: This method assembles individual objects at specific scales to reconstruct the original input frames.
  • Soft Z-buffer Rendering: A differentiable approach to handling occlusions that allows the system to efficiently evaluate thousands of potential scale combinations during training.

The training process follows a two-stage pipeline: an initial "bootstrapping" phase to establish basic object shapes, followed by an "alternative optimization" where the scale network and the object representations are iteratively refined.

Experimental Validation

We evaluated OSN across a diverse range of synthetic and real-world datasets, including the Dynamic Indoor Scene Dataset, the Oxford Multimotion Dataset, and the KITTI autonomous driving benchmark.

Embracing Ambiguity: The "Gnome House" Experiment

To demonstrate OSN's unique capabilities, we designed a benchmark featuring 50 distinct ground-truth 3D configurations that all produce the same 2D video. While baseline models struggled to produce even one accurate version, OSN successfully sampled and matched all 50 valid configurations, proving its ability to capture the full distribution of plausible scenes.

Quantitative Performance

OSN consistently outperforms state-of-the-art baselines such as NSFF, DynNeRF, HexPlane, and Total-Recon:

  • Novel View Synthesis: On the Indoor dataset, OSN achieved a PSNR of 25.98 dB, significantly higher than the next best competitor (24.69 dB).
  • Geometry Accuracy: The improvement in geometry is even more pronounced. OSN reached a SSIMAE (Scale and Shift-Invariant Mean Absolute Error) of 0.094, a massive leap over the 0.137 achieved by Total-Recon.

Qualitatively, OSN avoids common reconstruction artifacts where objects appear to "sink" into the ground or "float" unnaturally, thanks to its superior understanding of relative object scales and occlusions.

Limitations and Future Directions

While OSN represents a significant step forward, it currently focuses on rigid moving objects and does not yet handle deformable entities like walking people. Additionally, the method relies on external preprocessing for segmentation (SAM) and initial poses (SfM). Future work will aim to extend this "infinite representation" logic to deformable scenes and integrate preprocessing into an end-to-end framework.

By shifting the focus from finding a single solution to capturing the entire space of possibilities, OSN provides a more robust and accurate foundation for dynamic 3D scene understanding from monocular video.

Summary updated

Abstract

It has long been challenging to recover the underlying dynamic 3D scene representations from a monocular RGB video. Existing works formulate this problem into finding a single most plausible solution by adding various constraints such as depth priors and strong geometry constraints, ignoring the fact that there could be infinitely many 3D scene representations corresponding to a single dynamic video. In this paper, we aim to learn all plausible 3D scene configurations that match the input video, instead of just inferring a specific one. To achieve this ambitious goal, we introduce a new framework, called OSN. The key to our approach is a simple yet innovative object scale network together with a joint optimization module to learn an accurate scale range for every dynamic 3D object. This allows us to sample as many faithful 3D scene configurations as possible. Extensive experiments show that our method surpasses all baselines and achieves superior accuracy in dynamic novel view synthesis on multiple synthetic and real-world datasets. Most notably, our method demonstrates a clear advantage in learning fine-grained 3D scene geometry. Our code and data are available at https://github.com/vLAR-group/OSN
BibTeX
@inproceedings{song2024osn,
  author = {Ziyang Song and Jinxi Li and Bo Yang},
  title = {{OSN: Infinite Representations of Dynamic 3D Scenes from Monocular Videos}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=TTYVG17wfc},
}
Figure