PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets

Bi'an Du, Fanpeng Meng, Jiemin Fang, Chen Yang, Pufan Li, Daizong Liu, Qi Tian, Wei HuarXiv 2026 · arXiv

TL;DR

PhysAg3D is a new framework that turns a single image into a high-fidelity, physically interactive 3D asset by coupling 3D generative models with "agentic" reasoning. By using a vision-language agent to orchestrate geometric and physical tools, it produces assets that not only look like the original photograph but also feature plausible rigid articulation and compliant deformation.

What Problem Does This Paper Address?

We’ve become remarkably good at generating 3D meshes from single images, but these assets are usually "statues"—static shells with no internal logic for how they should move or deform.

The challenge is twofold:

  1. Underdetermination: A single image doesn't explicitly show where a joint is hidden or how "squishy" a material is.
  2. The Fidelity-Functionality Tension: Traditional methods that try to add interactivity often "break" the high-quality geometry or textures produced by state-of-the-art generators.

PhysAg3D addresses this by treating asset creation as a construction project managed by an intelligent agent. Instead of predicting a finished interactive model in one shot, it uses a generative model for the visual foundation and an agent to "build" and "test" the physical mechanisms.

Methods

The core of PhysAg3D is Generative-Agentic Coupling. The workflow follows three main stages:

  1. Foundation: A 3D generator (like TRELLIS.2) creates a high-fidelity textured mesh from the input image. Simultaneously, a Vision-Language Model (VLM) agent infers semantic parts and candidate interactions (e.g., "this is a laptop stand; the tray should tilt").
  2. Hierarchical Representation: The system uses a three-layer representation to link ideas to pixels:
  3. S (Interaction Intent): What the object should do (e.g., "lift arm").
  4. K (Physical Structure): The mechanical "skeleton" (e.g., a revolute joint with specific limits).
  5. X (Concrete Realization): The actual mesh partitions, materials, and "bindings" that connect the physical skeleton to the visible surface.
  6. Agentic Loop & Tool Use: The agent uses tools like Blender (for geometry/material edits) and MuJoCo (for physics simulation). Crucially, it performs feedback-guided refinement. If a simulated action fails or looks wrong, the agent attributes the error to a specific part of the hierarchy and tries a "repair" (e.g., adjusting a joint axis or re-partitioning a mesh).
Figure 1. Detailed appearance with region-specific interaction. Each example shows an input image, its generated asset, and either appearance details or a selected interaction state. (a) A camera shows detailed surface appearance; insets compare corresponding lens-barrel regions visible in the input image and generated asset. (b) A laptop stand adjusts its tray tilt and support-arm elevation. (c) Headphones combine a rigid support with locally compliant ear pads. These development examples show recorded simulated states in (b) and (c); ear-pad deformation uses an approximate compliance model.

Datasets

The authors evaluated PhysAg3D across several benchmarks:

  • In-the-wild Benchmark: A new collection of 50 real-world object photographs with 110 predefined semantic actions (e.g., opening a microwave, pressing a headphone pad).
  • PhysXVerse & PhysX-Mobility: Standard datasets used for reference-based evaluation of geometry and kinematics.
  • PhysX-Bench: A reference-free benchmark that uses automated "judges" to score visual quality and consistency.

Key Insights and Ideas

  • Appearance Preservation is Paramount: Unlike previous methods that might regenerate parts of a mesh (and lose texture continuity), PhysAg3D uses "bounded edits." It keeps the original UV maps and materials intact while only modifying the underlying structure needed for movement.
  • Intent vs. Implementation: By separating what should happen (Intent) from how it's built (Realization), the agent can fail at building a joint but still understand the goal, allowing for more effective troubleshooting during the refinement loop.
  • Rigid-Compliant Composition: The framework handles both "hard" parts (hinges on a laptop stand) and "soft" parts (deformable ear pads on headphones) within the same unified representation.
Figure 1. Examples of (a) photorealistic appearance, (b) physically plausible motion in a laptop stand, and (c) rigid-compliant composition in headphones.

Key Results

PhysAg3D significantly outperforms existing baselines (like aDSL and PhysX-Omni) in both visual fidelity and physical accuracy:

  • Action Completion: On the "in-the-wild" set, it achieved a 91.3% Action Completion Rate (ACR), compared to 66.8% for the strongest baseline.
  • Stability: It passed the Function-and-Visible-Stability Pass (FVSP) 88.0% of the time, whereas baselines often suffered from "exploding" physics or collapsed meshes (only 42.0% for aDSL).
  • Human Preference: In perceptual studies, humans rated PhysAg3D significantly higher for both input-image consistency (4.63/5) and motion plausibility (4.38/5).
Figure 4. Qualitative comparison on PhysXVerse showing how PhysAg3D preserves fine details (like patterns on headphones) during interaction compared to other methods.

Limitations and Future Work

  • Hidden Internal Structures: While the agent is good at inferring joints from external cues, it still struggles with complex internal mechanisms that are completely occluded in the source image.
  • Material Calibration: The current model uses "approximate compliance." Recovering exact physical parameters like Young's modulus or friction coefficients directly from a single 2D image remains an open challenge.
  • Computational Budget: The agentic loop (simulating, rendering, and reasoning) is more computationally expensive than simple feed-forward generation.

Summary updated

Abstract

Generating interactive 3D assets from a single image requires realistic appearance, fidelity to the depicted object, and plausible physical behavior. Yet detailed geometry alone does not determine how an object should move or deform. We introduce PhysAg3D, which couples 3D generative models with agentic reasoning and tool use. A vision-language agent first infers semantic parts, candidate interactions, and physical priors from the image; a 3D generator then produces a textured whole-object mesh from the same image. A hierarchical interaction representation links these revisable priors to mechanisms and the generated surfaces they control, organizing interaction construction and refinement. Guided by this representation and observations of the mesh, the agent uses geometric and physical tools to construct parts, joints, deformable regions, and surface bindings while retaining useful generated geometry and materials. Rendered interactions and execution measurements guide targeted revisions within a fixed budget. The framework accommodates both rigid articulation and compliant deformation. Quantitative and qualitative evaluations on real photographs and external benchmarks demonstrate detailed, image-aligned assets with plausible interactions. Ablations support bounded refinement and source-surface preservation. Project Page: supra-3d.github.io/PhysAg3D.
BibTeX
@misc{du2026physag,
  author = {Bi'an Du and Fanpeng Meng and Jiemin Fang and Chen Yang and Pufan Li and Daizong Liu and Qi Tian and Wei Hu},
  title = {{PhysAg3D: Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets}},
  year = {2026},
  eprint = {2610.39924},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39924},
}
Figure