TL;DR
TelePhysica is a closed-loop framework that transforms a single photograph into an interactive, physically-grounded 3D world. By using a vision-language coding agent and a two-stage verification loop (visual and physical), it ensures reconstructed scenes are stable under gravity and can drive highly realistic, counterfactual-aware video synthesis.
What Problem Does This Paper Address?
Current image-to-video (I2V) diffusion models are excellent at pixel synthesis but lack an internal "understanding" of physics. They often "hallucinate" motion that violates momentum balance, ignores object contact, or fails when asked to perform counterfactual actions (e.g., "what if I pulled the other corner of this towel?").
Simultaneously, traditional 3D reconstruction from a single image is "underconstrained." Because depth is ambiguous from one viewpoint, a reconstructed object might be placed slightly above a table or interpenetrating it. In a physics engine, this causes the scene to immediately explode or collapse. TelePhysica aims to bridge this gap by creating a persistent, executable 3D state that is both visually and physically verified.
Methods
The core of TelePhysica is an Agentic Reconstruction pipeline that follows a "Generate → Execute → Refine" loop:
- Vision-Language Coding Agent: An AI agent synthesizes a procedural scene program (meshes, materials, and physical properties) from a single image $I_0$.
- Two-Stage Verification Loop:
- Stage 1 (Visual Critique): Aligns rendered silhouettes with the original image using pixel-level mask-difference diagnostics.
- Stage 2 (Physical Preflight): The scene is compiled into a physics solver and integrated under gravity for 1.0s. If objects float, topple, or drift, the agent receives "drift traces" and must revise the geometry or poses.
- Unified Multi-Regime Solver: Unlike models that only handle rigid boxes, TelePhysica uses a solver that couples rigid bodies, articulated chains (e.g., earphone cables), thin-shell cloth (using XPBD), and 3D tetrahedral solids (for elastic foam).
- Trajectory-Grounded Video Synthesis: The verified 3D simulation generates "structural streams" (like Canny edges or shaded RGB renders) that condition a pretrained video diffusion model (e.g., Wan2.1-VACE). This ensures the final video pixels follow the laws of physics.
Datasets
- 100-Scene Tabletop Benchmark: A new benchmark introduced by the authors featuring rigid objects (vessels, stationery, produce, etc.) in complex arrangements.
- Complex Deformable Cases: Three high-fidelity reconstructions involving a microfiber bath towel, an upholstered chair cushion, and a stack of kitchen sponges.
- Evaluation: The model was tested against unconditioned I2V baselines (Direct I2V 1.3B and A14B).
Key Insights and Ideas
- The "Preflight" is Essential: Single-pass (open-loop) reconstructions had a 0% success rate in physical verification. Without the iterative loop, monocular depth errors always lead to physical instability.
- Physics as a Conditioning Stream: By providing a video model with a physically accurate "skeleton" of motion, the model can handle counterfactuals. For example, if you reverse a force vector, the simulation provides a new trajectory that the video model can actually follow, whereas standard I2V models often just repeat the same forward motion regardless of the prompt.
- Multi-Regime Fidelity: The authors show that treating a towel as a "rigid plate" proxy fails to capture realistic motion. Using co-dimensional finite-element cloth math results in 10x larger localized displacement and realistic buckling (pleating) that rigid proxies cannot replicate.
Key Results
- Recovery Rate: Closed-loop execution raised equilibrium-verified world recovery from 0% to 81% on the 100-scene benchmark.
- Motion Fidelity: Conditioning video diffusion on simulated trajectories increased dynamic motion-mask Intersection-over-Union (IoU) by 2×–5× compared to unconditioned baselines.
- Trajectory Accuracy: Centroid trajectory error was reduced by 4×–20×.
- Directional Alignment: Maintained > 0.90 directional cosine alignment under counterfactual force reversals, where standard models often yielded negative correlations (moving the wrong way).
Limitations and Future Work
- Parameter Priors: Currently, constitutive parameters like mass density, Young’s modulus (stiffness), and friction are assigned from material priors (e.g., "this looks like plastic, so use plastic values") rather than being estimated directly from the image.
- Future Direction: The authors suggest moving toward closed-loop physical system identification, where the agent could refine these physical parameters based on multi-step visual interaction feedback (e.g., "the object didn't slide as far as expected, so increase the friction coefficient").
Summary updated