Abstract
4D Primitive-Mâché (4DPM) introduces a novel framework for reconstructing complete, dynamic 3D scenes from single monocular RGB videos. By decomposing scenes into rigid 3D primitives and "glueing" their motions together through joint optimization, the system achieves "object permanence"—the ability to track and reconstruct objects even during temporary occlusions. Evaluated on the HO3D benchmark and a new multi-object dataset, 4DPM demonstrates significant improvements in reconstruction accuracy and temporal consistency over existing methods.
Introduction
Reconstructing a dynamic scene in 3D from a single moving camera remains a fundamental challenge in computer vision. While existing Simultaneous Localization and Mapping (SLAM) systems excel at static environments, and other methods can reconstruct visible parts of dynamic scenes, they often lack persistence. Most systems "forget" an object the moment it is occluded, such as when a toy is placed inside a drawer.
4DPM addresses this by building a persistent 4D map (3D geometry over time) that retains all observed information. This allows for a full "replay" of the scene where hidden objects continue to exist and move logically within the reconstruction, providing a form of spatial memory essential for advanced robotics and scene understanding.
Methodology
4DPM adopts a "piecewise-rigid" approach, simplifying complex per-pixel motion into the estimation of poses for discrete 3D "chunks" or primitives. The system operates through a frontend-backend architecture:
The Frontend: Decomposition
- Geometry: Initial 3D point maps for each frame are generated using a feed-forward model based on DUSt3R/$\pi^3$.
- Segmentation: The video is segmented into distinct object-like regions using SAMv2.
- Correspondence: A dense point tracker (AllTracker) establishes how pixels move between frames.
The Backend: Optimization and Persistence
- The "Glue": Each segment is treated as a rigid 3D primitive. The system performs a joint optimization to find the optimal $SE(3)$ poses (rotation and translation) that align these primitives across all timestamps based on 2D tracking data.
- Object Permanence: When an object becomes occluded, the system identifies "parent" objects in spatial contact or with similar velocities. It then extrapolates the hidden object's motion based on its parent, maintaining a persistent 3D representation even when the object is invisible to the camera.
Results and Evaluation
The authors evaluated 4DPM on two primary benchmarks: the standard HO3D dataset (hands manipulating objects) and a new Multi-Object Dataset collected using synchronized Azure Kinect cameras to provide high-quality multi-view ground truth for complex rigid motions.
Key Findings
- Accuracy: On the HO3D dataset, 4DPM achieved an F-score of 0.757, substantially outperforming TraceAnything (0.636) and POMATO (0.521).
- Occlusion Handling: The system successfully demonstrated object permanence in complex sequences, such as a closing drawer. By grouping the motion of internal objects with the drawer's body, 4DPM correctly inferred their positions after they were hidden.
- Efficiency: The backend performs dense alignment in approximately 10 seconds for a 512x512 resolution sequence, proving significantly faster than standard first-order optimizers.
Discussion and Limitations
4DPM demonstrates that 4D reconstruction is as much about "spatial memory" as it is about geometry. By reducing dimensionality through primitives, the optimization becomes tractable for real-world scenes. However, the system currently relies on a rigidity constraint, which limits its effectiveness with non-rigid deformations like human posture changes or soft object manipulation.
Future work aims to address the segmentation bottleneck—currently the slowest part of the pipeline—and transition the system from offline processing to incremental, real-time mapping for robotics applications.
Summary updated