Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraints

Lijun Guo, Haoyu Zhao, Xingyue Zhao, Rong Fu, Linghao Zhuang, Siteng Huang, Zhongyu Li, Hua ZouarXiv 2026 · arXiv · OpenReview

TL;DR

Articulat3D reconstructs detailed 3D "digital twins" of articulated objects (like doors or robotic arms) from single, casual videos, even when the action starts immediately. It achieves this by combining a novel motion prior initialization with strict geometric and kinematic constraints, outperforming previous methods in accuracy and real-world applicability.

What Problem Does This Paper Address?

Creating accurate, interactive 3D models of articulated objects, often called "digital twins," is crucial for applications like augmented reality and robotics. However, existing methods typically require multiple, carefully captured views of an object in distinct poses, which is impractical for real-world scenarios where objects are observed in casual, unconstrained videos. These methods also struggle with objects that start moving immediately, as they often rely on an initial static scan. The core challenge is to reconstruct both the object's geometry and its complex articulation from a single, potentially messy, monocular video without needing controlled capture.

Methods

Articulat3D tackles this by employing a two-stage optimization pipeline:

  1. Motion Prior-Driven Initialization: Instead of directly optimizing per-frame motion, this stage leverages 3D point tracks (obtained from tools like TAPIP3D) to infer a low-dimensional structure of articulated motion. It models the scene's dynamics using a compact set of "motion bases," allowing for a soft decomposition of the object into multiple moving parts. This stage provides a coherent, part-aware motion prior.
  2. Geometric and Motion Constraints Refinement: Building on the initialization, this stage enforces physically plausible articulation. It uses learnable "kinematic primitives" – specifically, joint axes, pivot points, and per-frame motion scalars – to define how parts move. These constraints are optimized to ensure that the reconstructed object's motion is not only geometrically accurate but also temporally coherent and adheres to rigid-body physics. This is a key departure from methods that optimize motion frame-by-frame independently.

Datasets

The paper uses and introduces several datasets:

  • Video2Articulation-S: A benchmark for simple articulated objects with single movable parts.
  • Articulat3D-Sim: A newly introduced dataset featuring complex multi-part kinematics (2-7 parts) from the PartNet-Mobility dataset, designed to simulate natural interactions with significant camera motion.
  • Articulat3D-Real: A new dataset of real-world sequences captured with an iPhone, featuring diverse lighting, clutter, and hand occlusions, to test generalizability.
  • VideoArtGS-20: An existing benchmark used for evaluating performance on a different dataset.

Key Insights and Ideas

  • Motion as a Continuous Trajectory, Not Discrete Frames: The core idea is to model articulated motion as a continuous, constrained process rather than a series of independent frame-wise optimizations. This is crucial for capturing smooth transitions and handling videos that start with motion.
  • Low-Dimensional Motion Bases for Initialization: Exploiting the fact that articulated motion often lies in a low-dimensional subspace significantly regularizes the problem and provides a robust starting point, even with noisy input.
  • Kinematic Primitives for Physical Plausibility: Explicitly enforcing learned kinematic constraints (joint axes, pivot points) is essential for creating "digital twins" that are not just visually accurate but also physically realistic and controllable.
  • Robustness to Noisy Input: The method demonstrates remarkable robustness to errors in initial 3D tracks and even to simulated noise, highlighting the strength of the multi-stage optimization and the learned constraints.

Key Results

  • State-of-the-Art Performance: Articulat3D consistently outperforms existing methods across synthetic and real-world benchmarks in joint estimation, reconstruction fidelity, tracking, and view synthesis.
  • High Accuracy: Achieves an average axis error of 1.60° on Video2Articulation-S and 0.53° on Articulat3D-Sim. Maintains sub-centimeter accuracy for reconstruction.
  • Efficiency: While requiring moderate optimization time, it offers a significant leap in fidelity compared to other methods, and is more efficient than some rendering-based approaches.
  • Qualitative Success: Visual results show accurate reconstruction of complex multi-part kinematics, preservation of structural rigidity, and robustness to challenging real-world conditions like hand occlusions and camera jitter.

Limitations and Future Work

  • Texture-less Surfaces: Performance may degrade on objects with very little texture, where obtaining reliable 3D tracks is difficult.
  • Independent Joints: The current implementation assumes independent joints, limiting the modeling of hierarchical kinematic structures (e.g., nested assemblies).
  • Future Work: The authors suggest integrating multi-view geometric priors and exploring the estimation of dynamic physical parameters like mass and friction to further enhance the digital twins.

Summary updated

Abstract

Building high-fidelity digital twins of articulated objects from visual data remains a central challenge. Existing approaches depend on multi-view captures of the object in discrete, static states, which severely constrains their real-world scalability. In this paper, we introduce Articulat3D, a novel framework that constructs such digital twins from casually captured monocular videos by jointly enforcing explicit 3D geometric and motion constraints. We first propose Motion Prior-Driven Initialization, which leverages 3D point tracks to exploit the low-dimensional structure of articulated motion. By modeling scene dynamics with a compact set of motion bases, we facilitate soft decomposition of the scene into multiple rigidly moving groups. Building on this initialization, we introduce Geometric and Motion Constraints Refinement, which enforces physically plausible articulation through learnable kinematic primitives parameterized by a joint axis, a pivot point, and per-frame motion scalars, yielding reconstructions that are both geometrically accurate and temporally coherent. Extensive experiments demonstrate that Articulat3D achieves state-of-the-art performance on synthetic benchmarks and real-world casually captured monocular videos, significantly advancing the feasibility of digital twin creation under uncontrolled real-world conditions. Our project page is available at https://maxwell-zhao.github.io/Articulat3D/.
BibTeX
@misc{guo2026articulat,
  author = {Lijun Guo and Haoyu Zhao and Xingyue Zhao and Rong Fu and Linghao Zhuang and Siteng Huang and Zhongyu Li and Hua Zou},
  title = {{Articulat3D: Reconstructing Articulated Digital Twins From Monocular Videos with Geometric and Motion Constraints}},
  year = {2026},
  eprint = {2603.11606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.11606},
}
Figure