TL;DR
$\pi^4$ is a unified foundation model that reconstructs dense 3D geometry and tracks motion from diverse RGB inputs—ranging from a single image to thousands of video frames—in a single forward pass. Despite having only 0.5B parameters (less than half the size of many baselines), it achieves state-of-the-art results across geometry benchmarks and maintains high surface consistency for 4D tracking.
What Problem Does This Paper Address?
Reconstructing a 3D scene and understanding how objects move within it (4D perception) usually requires complex, multi-stage pipelines. Traditional Structure-from-Motion (SfM) and Multi-View Stereo (MVS) methods are computationally heavy and often struggle with "in-the-wild" data where camera calibrations are unknown and scene content is unconstrained.
While recent feed-forward neural networks have made strides in geometry or motion estimation, they are often specialized. A major challenge remains: how do we build a single, efficient model that handles varying numbers of images, recovers metric scale, and ensures that motion tracks actually stay "registered" to the reconstructed 3D surfaces?
Methods
$\pi^4$ (pronounced "Pi-4") builds on the permutation-equivariant architecture of $\pi^3$ but extends it to handle motion and temporal dynamics. The architecture consists of three main components:
- Shared Backbone: A DINOv3 encoder extracts features from all input images.
- Looped Geometry Branch: This branch uses a Transformer decoder that shares weights across multiple "loops" or iterations. It treats all views symmetrically (no privileged reference view) to predict dense local pointmaps, camera poses, and a global metric scale. A sparse 3D refiner is used to sharpen local surface details.
- Geometry-Conditioned Motion Branch: This is the "secret sauce." Instead of predicting 3D motion in a vacuum, this branch is conditioned on the geometry features via Spatial Adaptive Layer Normalization (AdaLN). It predicts 2D image-space tracks and moving-object masks.
A key design choice is Lifting to 3D. Rather than regressing 3D coordinates directly—which often leads to tracks "floating" away from surfaces—$\pi^4$ predicts 2D correspondences and then samples the reconstructed 3D geometry at those locations. This ensures that every visible 3D track point lies exactly on the reconstructed scene surface.
Datasets
The authors trained $\pi^4$ on a massive mixture of over 40 datasets. Notable contributions to their data pipeline include:
- Scene-Level Synthetic Data: 10,000 assets from Funes and Sketchfab, rendered with "coverage-aware" view selection to ensure diverse camera angles and focal lengths.
- Foreground-Suppressed Internet Videos: To learn background geometry from videos dominated by moving objects, they used generative inpainting to "remove" foreground objects, creating clean pseudo-labels for training.
- GameRealm: A new dataset developed with collaborators that provides engine-native 3D geometry and persistent point trajectories from video games.
For evaluation, they tested against nine baselines across 16 settings, including standard benchmarks like Sintel, KITTI, and NYU-v2, as well as the long-sequence Oxford Spires dataset.
Key Insights and Ideas
- Parameter Efficiency via Looping: By reusing Transformer weights across decoding steps, the model stays compact (~0.5B parameters) while gaining the representational power of a much deeper network.
- Geometry as Context: The authors found that conditioning the motion branch on geometry features reduces endpoint error (EPE) by over 7 pixels, particularly in pairs with large camera baselines. Geometry provides the necessary spatial "roadmap" for motion.
- Surface-Constrained Tracking: The decision to "lift" 2D tracks to the 3D surface (rather than regressing 3D directly) is a major win for qualitative consistency. It prevents the "drifting" common in other 4D models.
- Handling Scale: The model produces joint reconstructions for scenes with over 1,000 images in one go, even though it was trained on contexts of at most 64 views.
Key Results
- Geometry Champion: $\pi^4$ achieved the best overall average rank across 16 geometry benchmarks, outperforming larger models like Pi3X and VGGT.
- Long-Sequence Performance: On the Oxford Spires dataset (361–800 frames), $\pi^4$ reduced the Absolute Trajectory Error (ATE) by 47.2% compared to the previous best model (DVLT).
- Efficiency: It produces joint reconstructions and tracks in a single forward pass, making it significantly faster and more practical for large-scale applications than iterative optimization methods.
Limitations and Future Work
- Motion Ambiguity: In scenes with weak texture or dominant foregrounds, the model can sometimes confuse camera motion with object motion.
- Coordinate Sensitivity: Accuracy tends to degrade for cameras located far from the reconstruction's coordinate origin.
- The Optimization Gap: While $\pi^4$ is the best feed-forward model, it still doesn't quite match the global consistency of traditional, slow geometric optimization methods on extremely long sequences. Closing this gap is the next frontier.
For those interested in exploring further, the code and models are available at Pi4.
Summary updated