TelePhysica: Agentic Reconstruction of Physically Interactive Scenes

Ruiqi Shu, Xufeng Zhan, Yuanzhi Liang, Haibin Huang, Chi Zhang, Xuelong LiarXiv 2026 · arXiv

TL;DR

TelePhysica is a closed-loop framework that transforms a single photograph into an interactive, physically-grounded 3D world. By using a vision-language coding agent and a two-stage verification loop (visual and physical), it ensures reconstructed scenes are stable under gravity and can drive highly realistic, counterfactual-aware video synthesis.

What Problem Does This Paper Address?

Current image-to-video (I2V) diffusion models are excellent at pixel synthesis but lack an internal "understanding" of physics. They often "hallucinate" motion that violates momentum balance, ignores object contact, or fails when asked to perform counterfactual actions (e.g., "what if I pulled the other corner of this towel?").

Simultaneously, traditional 3D reconstruction from a single image is "underconstrained." Because depth is ambiguous from one viewpoint, a reconstructed object might be placed slightly above a table or interpenetrating it. In a physics engine, this causes the scene to immediately explode or collapse. TelePhysica aims to bridge this gap by creating a persistent, executable 3D state that is both visually and physically verified.

Methods

The core of TelePhysica is an Agentic Reconstruction pipeline that follows a "Generate → Execute → Refine" loop:

  1. Vision-Language Coding Agent: An AI agent synthesizes a procedural scene program (meshes, materials, and physical properties) from a single image $I_0$.
  2. Two-Stage Verification Loop:
    • Stage 1 (Visual Critique): Aligns rendered silhouettes with the original image using pixel-level mask-difference diagnostics.
  3. Stage 2 (Physical Preflight): The scene is compiled into a physics solver and integrated under gravity for 1.0s. If objects float, topple, or drift, the agent receives "drift traces" and must revise the geometry or poses.
  4. Unified Multi-Regime Solver: Unlike models that only handle rigid boxes, TelePhysica uses a solver that couples rigid bodies, articulated chains (e.g., earphone cables), thin-shell cloth (using XPBD), and 3D tetrahedral solids (for elastic foam).
  5. Trajectory-Grounded Video Synthesis: The verified 3D simulation generates "structural streams" (like Canny edges or shaded RGB renders) that condition a pretrained video diffusion model (e.g., Wan2.1-VACE). This ensures the final video pixels follow the laws of physics.
Figure 1: TelePhysica overview. (a) Closed-loop recovery from a single photograph 𝐼0 refines a procedural scene program via Stage-1 silhouette critique and Stage-2 gravitational equilibrium checks. (b) Unified stateful mechanics spans SE(3) rigid contact, 164-link chains, thin-shell cloth, and 3D tetrahedral solids under surface wrenches 𝝉(𝑡) = r(𝑡) × F(𝑡) and patch tractions fext,𝑎(𝑡) = 𝑤𝑎Fpatch𝑤(𝑡). (c) Phase-aligned physics and VACE frames across rigid tumbling (A14B RGB), cloth folding (A14B RGB), and solid indentation (A14B Canny).

Datasets

  • 100-Scene Tabletop Benchmark: A new benchmark introduced by the authors featuring rigid objects (vessels, stationery, produce, etc.) in complex arrangements.
  • Complex Deformable Cases: Three high-fidelity reconstructions involving a microfiber bath towel, an upholstered chair cushion, and a stack of kitchen sponges.
  • Evaluation: The model was tested against unconditioned I2V baselines (Direct I2V 1.3B and A14B).

Key Insights and Ideas

  • The "Preflight" is Essential: Single-pass (open-loop) reconstructions had a 0% success rate in physical verification. Without the iterative loop, monocular depth errors always lead to physical instability.
  • Physics as a Conditioning Stream: By providing a video model with a physically accurate "skeleton" of motion, the model can handle counterfactuals. For example, if you reverse a force vector, the simulation provides a new trajectory that the video model can actually follow, whereas standard I2V models often just repeat the same forward motion regardless of the prompt.
  • Multi-Regime Fidelity: The authors show that treating a towel as a "rigid plate" proxy fails to capture realistic motion. Using co-dimensional finite-element cloth math results in 10x larger localized displacement and realistic buckling (pleating) that rigid proxies cannot replicate.
The two-stage verification process showing how visual and physical checks refine a scene.

Key Results

  • Recovery Rate: Closed-loop execution raised equilibrium-verified world recovery from 0% to 81% on the 100-scene benchmark.
  • Motion Fidelity: Conditioning video diffusion on simulated trajectories increased dynamic motion-mask Intersection-over-Union (IoU) by 2×–5× compared to unconditioned baselines.
  • Trajectory Accuracy: Centroid trajectory error was reduced by 4×–20×.
  • Directional Alignment: Maintained > 0.90 directional cosine alignment under counterfactual force reversals, where standard models often yielded negative correlations (moving the wrong way).
Comparison of TelePhysica's physics-guided video synthesis against direct image-to-video baselines.

Limitations and Future Work

  • Parameter Priors: Currently, constitutive parameters like mass density, Young’s modulus (stiffness), and friction are assigned from material priors (e.g., "this looks like plastic, so use plastic values") rather than being estimated directly from the image.
  • Future Direction: The authors suggest moving toward closed-loop physical system identification, where the agent could refine these physical parameters based on multi-step visual interaction feedback (e.g., "the object didn't slide as far as expected, so increase the friction coefficient").

Summary updated

Abstract

Direct image-to-video diffusion models entangle pixel synthesis with implicit latent motion, precluding persistent 3D mechanical equilibrium, mid-rollout intervention, and counterfactual force control. Conversely, single-pass programmatic 3D reconstruction from an uncalibrated photograph leaves monocular depth and support clearance underconstrained, causing reconstructed scenes to interpenetrate or collapse under gravity. We present TelePhysica, a closed-loop framework that reconstructs an executable, multi-regime physical world from a single photograph to drive stateful 3D interaction and trajectory-grounded video synthesis. A vision-language coding agent synthesizes separable visual meshes and declarative physical models within a two-stage verification loop that pairs pixel-aligned silhouette critique with a zero-force gravitational equilibrium preflight. The verified world compiles into a unified solver coupling rigid bodies, articulated chains, co-dimensional cloth shells, and 3D tetrahedral elastodynamics, whose simulated rollouts condition pretrained video diffusion models through pixel-aligned structural streams. On a 100-scene single-image benchmark, closed-loop execution raises equilibrium-verified world recovery from $0\%$ in single-pass synthesis to $81\%$. Controlled constitutive ablations show over an order-of-magnitude larger localized corner displacement and out-of-plane pleating in finite-element cloth than in matched rigid-plate proxies. Conditioning video diffusion on simulated trajectories increases dynamic motion-mask overlap by $2{\times}\text{--}5{\times}$ and preserves motion direction under counterfactual force reversals where direct image-to-video models fail.
BibTeX
@misc{shu2026telephysica,
  author = {Ruiqi Shu and Xufeng Zhan and Yuanzhi Liang and Haibin Huang and Chi Zhang and Xuelong Li},
  title = {{TelePhysica: Agentic Reconstruction of Physically Interactive Scenes}},
  year = {2026},
  eprint = {2611.18078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18078},
}
Figure