PhyVGGT: Feed-Forward Articulated Object Reconstruction from Monocular Videos

Qi Shao, Zhihao Cao, Shuhao Zhai, Anh Nguyen, Hong Liang, Baoru Huang, Liang LiarXiv 2026 · arXiv

TL;DR

PhyVGGT is a new feed-forward framework that reconstructs 3D articulated objects—including their movable parts, joint axes, and pivots—from a single uncalibrated monocular video in just seconds. By anchoring motion estimation to the stable geometry of an object's stationary base, it overcomes the depth biases and motion blur that typically plague monocular 3D reconstruction, achieving state-of-the-art performance and generalizing zero-shot from synthetic training to real-world scenes.

What Problem Does This Paper Address?

Reconstructing articulated objects (like cabinets, ovens, or tools) in 3D is a cornerstone of robotic manipulation and digital twin creation. However, doing this from a "casual" monocular video—without specialized depth sensors or pre-calculated camera poses—is incredibly difficult.

Existing solutions generally fall into two camps, both with significant drawbacks:

  1. Per-scene optimization: These methods (like Articulat3D) produce high-quality results but are painfully slow, often requiring minutes or hours of computation per video.
  2. Feed-forward foundation models: While models like VGGT can generate dense 3D point maps quickly, they treat scenes as static. They struggle to segment moving parts accurately because cross-frame attention tends to blur boundaries, and they often underestimate motion along the camera's viewing ray (depth bias).

PhyVGGT aims to provide the best of both worlds: the speed of a feed-forward model with the kinematic accuracy required for real-world interaction.

Methods

PhyVGGT is built on top of a frozen 3D foundation model (VGGT). The authors introduce two primary modules to handle articulation:

1. Part and Motion Decoder

This module fuses intermediate per-frame tokens with global tokens to maintain sharp boundaries for moving parts. Crucially, it uses a static-base query. By referencing all moving part queries to this stationary anchor, the system can cancel out camera drift and align 3D point maps across frames even when the camera is moving.

2. Kinematic Inference Layer

Instead of trying to regress joint parameters in a continuous 3D space (which is often ill-conditioned), this layer exploits the structural regularity of man-made objects.

  • Manhattan Frame Fitting: It fits an orthogonal "Manhattan frame" to the surface normals of the stationary base. Since most hinges on furniture and appliances align with the object's primary axes, this restricts the search for the joint axis to three orthogonal candidates.
  • Bingham Posteriors: It uses a Bingham distribution—a statistical tool for modeling orientations—to combine observed motion evidence with learned priors. This allows the model to select the correct joint axis in a closed-form, discrete manner, which is much more robust than continuous regression.
  • Linear Pivot Solver: Finally, it solves a positive-definite linear system to find the hinge pivot, allowing the model to "drive" the object into unseen states (e.g., imagining a drawer fully open when the video only showed it halfway).
Fig. 1. Overview of single-pass 3D articulated object reconstruction from uncalibrated monocular RGB videos. Given casual RGB frames (left), PhyVGGT segments 3D movable parts (teal), estimates joint axes (orange) and hinge pivots (red) across observed articulation states, and drives each part to unseen states in 1.2 s per clip (centre), generalising zero-shot from synthetic training to diverse real-world scenes (right).

Datasets

The model is trained entirely on synthetic data from PartNet-Mobility, featuring thousands of articulated objects in simulated indoor environments.

For evaluation, the authors tested zero-shot generalization on several challenging benchmarks:

  • Arti4D: Egocentric handheld videos of doors and appliances.
  • D3D-HOI: Third-person videos of human-object interactions.
  • RBO: Real-world videos with motion-capture ground truth.
  • Articulat3D-Sim: A synthetic benchmark for long-sequence reconstruction.

Key Insights and Ideas

  • The "Static Anchor" Strategy: The most significant insight is that while moving parts are distorted by monocular depth bias, the stationary base remains geometrically consistent. By anchoring all kinematic inferences to the base's coordinate system, the model bypasses much of the noise inherent in monocular video.
  • Discrete Selection over Continuous Regression: Estimating a 3D vector for a joint axis is hard. Choosing which of the three Manhattan axes is the "hinge" is much easier and more robust to the "tilt" caused by depth errors.
  • Zero-Shot Sim-to-Real: The reliance on geometric primitives (like Manhattan frames) rather than just raw appearance allows the model to transfer seamlessly from synthetic training to diverse real-world videos without fine-tuning.
Architecture diagram showing the Part and Motion Decoder and Kinematic Inference Layer.

Key Results

  • Speed: PhyVGGT processes a 48-frame video in 6.8 seconds, a massive speedup compared to the hours required by optimization-based methods like Articulat3D (over 1,300x faster).
  • Accuracy: On the Arti4D benchmark, it achieved a median axis error of 4.79°, significantly outperforming other monocular methods like FAMOS (23.24°) and even surpassing some methods that use privileged RGB-D data.
  • Robustness: The model successfully reconstructed objects even under small opening angles where traditional motion-fitting methods typically fail.

Limitations and Future Work

The authors note that because the system relies on passive RGB video, it cannot "see" through occlusions or infer hidden internal parts that don't move during the clip. Furthermore, it reconstructs geometry and kinematics but does not yet estimate physical properties like mass, friction, or the presence of springs and latches. The authors suggest that future work could involve interactive manipulation to recover these hidden physical parameters.

Summary updated

Abstract

Reconstructing an articulated object with its 3D movable parts, joint parameters and unseen articulation states from an uncalibrated monocular video is essential for robot manipulation and interactive digital twins. Existing per-scene optimisation methods require minutes to hours and external depth or camera poses, whereas feed-forward 3D foundation models reconstruct dense point maps in one pass without segmenting movable parts or inferring joints. Even on these monocular point maps, regressing or fitting continuous joints directly fails on casual videos, since cross-frame attention blurs moving part boundaries in deep layers, underestimates motion along the viewing ray and leaves continuous axes ill-conditioned under small opening angles. To address these challenges, we propose PhyVGGT, a single-pass framework on a frozen 3D foundation model that anchors moving-part matching and closed-form joint estimation to the undistorted geometry of the stationary base. First, our Part and Motion Decoder fuses intermediate per-frame tokens with global tokens and references part queries to a static-base query to segment 3D parts, align point maps on the static base and predict learned Bingham priors. Second, our Kinematic Inference Layer fits an orthogonal Manhattan frame to static surface normals, selects the joint axis in closed form under a Bingham posterior combining pairwise motion scatter with the learned priors, and solves a linear system for the hinge pivot to drive unseen states. Trained solely on synthetic data, PhyVGGT achieves state-of-the-art monocular performance across five benchmarks, outperforming FAMOS and monocular ArtiPoint, surpassing posed RGB-D ArtiPoint in median error, and running in 6.8 s per 48-frame video against hours of per-scene optimisation in Articulat3D. Project page: https://phyvggt.github.io.
BibTeX
@misc{shao2026phyvggt,
  author = {Qi Shao and Zhihao Cao and Shuhao Zhai and Anh Nguyen and Hong Liang and Baoru Huang and Liang Li},
  title = {{PhyVGGT: Feed-Forward Articulated Object Reconstruction from Monocular Videos}},
  year = {2026},
  eprint = {2611.22244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22244},
}
Figure