Monocular 4D reconstruction—the task of recovering 3D geometry and camera motion from a single video—remains a fundamental challenge in computer vision. While traditional methods like Structure-from-Motion (SfM) excel in static environments, they often falter in dynamic "in-the-wild" scenes where objects move independently. Geo4D introduces a novel paradigm: repurposing the rich, implicit world priors found in large-scale video diffusion models to perform explicit geometric reconstruction.
Methodology: From Video Generation to Geometric Inference
The core innovation of Geo4D lies in transforming a pre-trained video generator (specifically DynamiCrafter) into a multi-modal geometric predictor. Rather than estimating depth alone, the model predicts three complementary geometric modalities:
- Point Maps: Viewpoint-invariant 3D coordinates relative to a reference frame, crucial for global alignment.
- Disparity Maps: Inverse depth values that capture fine-grained local geometry.
- Ray Maps: Encodings of camera orientation and intrinsics for every pixel.
The system processes video in sliding windows, using a U-Net backbone to denoise geometric features. To ensure temporal consistency across these windows, Geo4D employs a Multi-Modal Alignment algorithm. This global optimization step reconciles the redundant information from point, disparity, and ray maps, weighted by predicted uncertainty maps to prioritize stable scene structures over transient or ambiguous regions.
Key Results and Generalization
Despite being trained exclusively on synthetic datasets (such as PointOdyssey and TartanAir), Geo4D demonstrates remarkable zero-shot generalization to real-world footage.
Quantitative Performance
On the Sintel benchmark, Geo4D achieved a state-of-the-art Absolute Relative Error (Abs Rel) of 0.205, representing a 24% improvement over previous diffusion-based depth models like DepthCrafter. It also significantly outperforms discriminative models in camera pose estimation, particularly in high-motion scenarios.
Qualitative Superiority
Geo4D excels in handling complex visual phenomena that typically confuse monocular models. For instance, it correctly interprets reflections in water as non-solid geometry and maintains structural integrity during rapid camera pans.
Limitations and Future Directions
While Geo4D sets a new benchmark for monocular 4D reconstruction, certain challenges remain. The accuracy of the final reconstruction is currently bounded by the resolution of the VAE bottleneck. Additionally, the model can struggle with extreme focal length changes (e.g., dolly zooms) not well-represented in synthetic training data. Future work aims to extend this framework to multi-view configurations, potentially creating a truly universal 4D foundation model.
Summary updated