$π^4$: A Foundation Model for 4D Scene Geometry and Motion

Yifan Wang, Weiguang Zhao, Ni Yang, Haoyu Guo, Junting Dong, Weinan Zhang, Chunhua ShenarXiv 2026 · arXiv

TL;DR

$\pi^4$ is a unified foundation model that reconstructs dense 3D geometry and tracks motion from diverse RGB inputs—ranging from a single image to thousands of video frames—in a single forward pass. Despite having only 0.5B parameters (less than half the size of many baselines), it achieves state-of-the-art results across geometry benchmarks and maintains high surface consistency for 4D tracking.

What Problem Does This Paper Address?

Reconstructing a 3D scene and understanding how objects move within it (4D perception) usually requires complex, multi-stage pipelines. Traditional Structure-from-Motion (SfM) and Multi-View Stereo (MVS) methods are computationally heavy and often struggle with "in-the-wild" data where camera calibrations are unknown and scene content is unconstrained.

While recent feed-forward neural networks have made strides in geometry or motion estimation, they are often specialized. A major challenge remains: how do we build a single, efficient model that handles varying numbers of images, recovers metric scale, and ensures that motion tracks actually stay "registered" to the reconstructed 3D surfaces?

Methods

$\pi^4$ (pronounced "Pi-4") builds on the permutation-equivariant architecture of $\pi^3$ but extends it to handle motion and temporal dynamics. The architecture consists of three main components:

  1. Shared Backbone: A DINOv3 encoder extracts features from all input images.
  2. Looped Geometry Branch: This branch uses a Transformer decoder that shares weights across multiple "loops" or iterations. It treats all views symmetrically (no privileged reference view) to predict dense local pointmaps, camera poses, and a global metric scale. A sparse 3D refiner is used to sharpen local surface details.
  3. Geometry-Conditioned Motion Branch: This is the "secret sauce." Instead of predicting 3D motion in a vacuum, this branch is conditioned on the geometry features via Spatial Adaptive Layer Normalization (AdaLN). It predicts 2D image-space tracks and moving-object masks.
The $\pi^4$ architecture overview, showing the shared encoder and the two specialized looped decoders.

A key design choice is Lifting to 3D. Rather than regressing 3D coordinates directly—which often leads to tracks "floating" away from surfaces—$\pi^4$ predicts 2D correspondences and then samples the reconstructed 3D geometry at those locations. This ensures that every visible 3D track point lies exactly on the reconstructed scene surface.

Datasets

The authors trained $\pi^4$ on a massive mixture of over 40 datasets. Notable contributions to their data pipeline include:

  • Scene-Level Synthetic Data: 10,000 assets from Funes and Sketchfab, rendered with "coverage-aware" view selection to ensure diverse camera angles and focal lengths.
  • Foreground-Suppressed Internet Videos: To learn background geometry from videos dominated by moving objects, they used generative inpainting to "remove" foreground objects, creating clean pseudo-labels for training.
  • GameRealm: A new dataset developed with collaborators that provides engine-native 3D geometry and persistent point trajectories from video games.

For evaluation, they tested against nine baselines across 16 settings, including standard benchmarks like Sintel, KITTI, and NYU-v2, as well as the long-sequence Oxford Spires dataset.

Key Insights and Ideas

  • Parameter Efficiency via Looping: By reusing Transformer weights across decoding steps, the model stays compact (~0.5B parameters) while gaining the representational power of a much deeper network.
  • Geometry as Context: The authors found that conditioning the motion branch on geometry features reduces endpoint error (EPE) by over 7 pixels, particularly in pairs with large camera baselines. Geometry provides the necessary spatial "roadmap" for motion.
  • Surface-Constrained Tracking: The decision to "lift" 2D tracks to the 3D surface (rather than regressing 3D directly) is a major win for qualitative consistency. It prevents the "drifting" common in other 4D models.
  • Handling Scale: The model produces joint reconstructions for scenes with over 1,000 images in one go, even though it was trained on contexts of at most 64 views.
Qualitative results showing geometry and motion tracks for various complex scenes.

Key Results

  • Geometry Champion: $\pi^4$ achieved the best overall average rank across 16 geometry benchmarks, outperforming larger models like Pi3X and VGGT.
  • Long-Sequence Performance: On the Oxford Spires dataset (361–800 frames), $\pi^4$ reduced the Absolute Trajectory Error (ATE) by 47.2% compared to the previous best model (DVLT).
  • Efficiency: It produces joint reconstructions and tracks in a single forward pass, making it significantly faster and more practical for large-scale applications than iterative optimization methods.

Limitations and Future Work

  • Motion Ambiguity: In scenes with weak texture or dominant foregrounds, the model can sometimes confuse camera motion with object motion.
  • Coordinate Sensitivity: Accuracy tends to degrade for cameras located far from the reconstruction's coordinate origin.
  • The Optimization Gap: While $\pi^4$ is the best feed-forward model, it still doesn't quite match the global consistency of traditional, slow geometric optimization methods on extremely long sequences. Closing this gap is the next frontier.

For those interested in exploring further, the code and models are available at Pi4.

Summary updated

Abstract

$π^4$ is a unified foundation model for 4D scene perception. From diverse, uncalibrated RGB observations in the wild---a single image, an unordered collection, or a video sequence---it recovers dense geometry, camera poses, and metric scale, together with 2D/3D point tracks and moving-object masks for multi-image inputs. It couples a geometry branch with a geometry-conditioned motion branch, which predicts image-space correspondences that are lifted into 3D using reconstructed target-view geometry. Across diverse benchmarks, $π^4$ achieves the best overall average rank in geometry evaluation, alongside competitive performance in single- and multiview 2D/3D point tracking and moving-object segmentation. With a shared backbone, the full geometry-and-motion system contains approximately 0.5B parameters---less than half the size of most baselines. It produces joint reconstructions and tracks in one forward pass, including large-scale scenes with thousands of input images despite training with contexts of at most 64 views. We further present the data and training recipe underlying these capabilities, together with empirical studies of key design choices. Code and models are available at https://github.com/yyfz/Pi4.
BibTeX
@misc{wang2026foundation,
  author = {Yifan Wang and Weiguang Zhao and Ni Yang and Haoyu Guo and Junting Dong and Weinan Zhang and Chunhua Shen},
  title = {{$π^4$: A Foundation Model for 4D Scene Geometry and Motion}},
  year = {2026},
  eprint = {2611.23144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23144},
}
Figure