TL;DR
PhyVGGT is a new feed-forward framework that reconstructs 3D articulated objects—including their movable parts, joint axes, and pivots—from a single uncalibrated monocular video in just seconds. By anchoring motion estimation to the stable geometry of an object's stationary base, it overcomes the depth biases and motion blur that typically plague monocular 3D reconstruction, achieving state-of-the-art performance and generalizing zero-shot from synthetic training to real-world scenes.
What Problem Does This Paper Address?
Reconstructing articulated objects (like cabinets, ovens, or tools) in 3D is a cornerstone of robotic manipulation and digital twin creation. However, doing this from a "casual" monocular video—without specialized depth sensors or pre-calculated camera poses—is incredibly difficult.
Existing solutions generally fall into two camps, both with significant drawbacks:
- Per-scene optimization: These methods (like Articulat3D) produce high-quality results but are painfully slow, often requiring minutes or hours of computation per video.
- Feed-forward foundation models: While models like VGGT can generate dense 3D point maps quickly, they treat scenes as static. They struggle to segment moving parts accurately because cross-frame attention tends to blur boundaries, and they often underestimate motion along the camera's viewing ray (depth bias).
PhyVGGT aims to provide the best of both worlds: the speed of a feed-forward model with the kinematic accuracy required for real-world interaction.
Methods
PhyVGGT is built on top of a frozen 3D foundation model (VGGT). The authors introduce two primary modules to handle articulation:
1. Part and Motion Decoder
This module fuses intermediate per-frame tokens with global tokens to maintain sharp boundaries for moving parts. Crucially, it uses a static-base query. By referencing all moving part queries to this stationary anchor, the system can cancel out camera drift and align 3D point maps across frames even when the camera is moving.
2. Kinematic Inference Layer
Instead of trying to regress joint parameters in a continuous 3D space (which is often ill-conditioned), this layer exploits the structural regularity of man-made objects.
- Manhattan Frame Fitting: It fits an orthogonal "Manhattan frame" to the surface normals of the stationary base. Since most hinges on furniture and appliances align with the object's primary axes, this restricts the search for the joint axis to three orthogonal candidates.
- Bingham Posteriors: It uses a Bingham distribution—a statistical tool for modeling orientations—to combine observed motion evidence with learned priors. This allows the model to select the correct joint axis in a closed-form, discrete manner, which is much more robust than continuous regression.
- Linear Pivot Solver: Finally, it solves a positive-definite linear system to find the hinge pivot, allowing the model to "drive" the object into unseen states (e.g., imagining a drawer fully open when the video only showed it halfway).
Datasets
The model is trained entirely on synthetic data from PartNet-Mobility, featuring thousands of articulated objects in simulated indoor environments.
For evaluation, the authors tested zero-shot generalization on several challenging benchmarks:
- Arti4D: Egocentric handheld videos of doors and appliances.
- D3D-HOI: Third-person videos of human-object interactions.
- RBO: Real-world videos with motion-capture ground truth.
- Articulat3D-Sim: A synthetic benchmark for long-sequence reconstruction.
Key Insights and Ideas
- The "Static Anchor" Strategy: The most significant insight is that while moving parts are distorted by monocular depth bias, the stationary base remains geometrically consistent. By anchoring all kinematic inferences to the base's coordinate system, the model bypasses much of the noise inherent in monocular video.
- Discrete Selection over Continuous Regression: Estimating a 3D vector for a joint axis is hard. Choosing which of the three Manhattan axes is the "hinge" is much easier and more robust to the "tilt" caused by depth errors.
- Zero-Shot Sim-to-Real: The reliance on geometric primitives (like Manhattan frames) rather than just raw appearance allows the model to transfer seamlessly from synthetic training to diverse real-world videos without fine-tuning.
Key Results
- Speed: PhyVGGT processes a 48-frame video in 6.8 seconds, a massive speedup compared to the hours required by optimization-based methods like Articulat3D (over 1,300x faster).
- Accuracy: On the Arti4D benchmark, it achieved a median axis error of 4.79°, significantly outperforming other monocular methods like FAMOS (23.24°) and even surpassing some methods that use privileged RGB-D data.
- Robustness: The model successfully reconstructed objects even under small opening angles where traditional motion-fitting methods typically fail.
Limitations and Future Work
The authors note that because the system relies on passive RGB video, it cannot "see" through occlusions or infer hidden internal parts that don't move during the clip. Furthermore, it reconstructs geometry and kinematics but does not yet estimate physical properties like mass, friction, or the presence of springs and latches. The authors suggest that future work could involve interactive manipulation to recover these hidden physical parameters.
Summary updated