PRIO: Pre-trained Representation for Interactive Objects in 4D Worlds

Shixun Huang, Lifu Wei, Yue Chang, Chen Geng, Heng Zhang, Jinhyung Ahn, Caoliwen Wang, Siyuan Chen, Donglai Xiang, Jiajun Wu, Peter Yichen ChenarXiv 2026 · arXiv

TL;DR

PRIO is a pre-trained model that transforms static 3D objects—whether meshes, point clouds, or 3D Gaussians—into interactive, physically responsive "4D" entities in a single forward pass. By predicting a linear configuration space, it enables real-time physics simulations that respond to user actions at over 30 FPS without requiring per-object training or expensive neural inference at every time step.

What Problem Does This Paper Address?

We are currently in an era where 3D worlds can be captured or generated at scale, but these worlds are largely "frozen." A reconstructed pillow doesn't compress when pressed, and a generated door doesn't swing when pulled.

Existing solutions to this "static world" problem fall into two camps, both with significant drawbacks:

  1. Generative Video Models: These produce convincing motion but output pixels rather than consistent 3D objects, and they require heavy computation for every single frame.
  2. Physics-Based Simulation: Traditional methods can be physically accurate but often take minutes or hours to compute. Recent "learned" simulators either require training a new model for every specific object or use non-linear spaces that require running a neural network at every step of the simulation, killing real-time performance.

PRIO bridges this gap by providing a representation that is both generalizable (works on unseen objects) and efficient (runs at interactive rates).

Methods

The core innovation of PRIO is predicting a linear configuration space for an object based solely on its static geometry.

Figure 2. Overview of PRIO. Given a mesh, point cloud or 3D Gaussians, PRIO encodes the geometry into shape tokens and predicts, at any point X, the fields ΦG(X) that span a linear configuration space. The network runs once per object; at inference, user actions drive the dynamics through the Euler–Lagrange equations in real time.

The Architecture

  1. Shape Encoding: PRIO samples points from the object and uses a DiffusionNet to extract local and global geometric features.
  2. Shape Tokens: These features are condensed into a set of "shape tokens" via self-attention, summarizing different regions of the object.
  3. Field Prediction: A decoder uses cross-attention between query points and these tokens to predict a continuous field, $\Phi_G(X)$. This field defines the "basis" of how the object can move and deform.
  4. Linearity: Because the deformation is modeled as a linear combination of these basis fields ($x = X + \Phi q$), the heavy lifting of the physics (the mass matrices and potential derivatives) can be precomputed once.

The Physics

Once the linear space is predicted, PRIO uses Lagrangian dynamics. User actions and physical laws (like gravity or elasticity) drive a small set of generalized coordinates ($q$). Because the mapping is linear, the simulation becomes a series of efficient matrix-vector multiplications that run entirely on the GPU without further network inference.

Datasets

  • Pre-training: PRIO is pre-trained on a diverse collection of simulated 4D trajectories (deforming meshes and objects).
  • Evaluation:
    • Deform360: A dataset of real-world objects deforming under user interaction, used to test zero-shot transfer to real captures.
  • Unseen Shapes: Various 3D models not present in the training set to test generalization.
  • Benchmarks: Compared against specialized rigging and skinning methods like PhysSkin, UniRig, and NeuROK.

Key Insights and Ideas

  • Linearity is a Feature, Not a Bug: While non-linear spaces can be more expressive, they are computationally expensive. PRIO shows that a well-predicted linear space is sufficient for high-quality, real-time interaction.
  • Encoder-Free Training: Instead of training an encoder to find latent coordinates, PRIO solves for the optimal coordinates during training using a closed-form least-squares solution (the pseudo-inverse). This ensures the model learns the best possible configuration space the decoder can represent.
  • Format Agnostic: Because PRIO predicts a continuous field, it doesn't care if your input is a mesh, a point cloud, or 3D Gaussians. It can drive the "interactive 4D" version of any of them.
PRIO enables real-time interaction across different formats, including 3D Gaussians and point clouds.

Key Results

  • Superior Accuracy: In inverse kinematics tasks (fitting the model to a target shape), PRIO outperformed the next best method (PhysSkin) by a significant margin, reducing Chamfer $L_1$ error from 0.01299 to 0.00522.
  • Speed: PRIO runs 7.1x faster than Vertex Block Descent (VBD) for similar accuracy levels.
  • Real-Time Performance: The system consistently runs at interactive rates. For example, it can simulate a scene with eight plush toys (nearly 1M triangles) at 25ms per step on a single GPU.
  • Physical Plausibility: Using Vision-Language Model (VLM) scoring, PRIO was rated significantly higher than generative baselines for physical common sense and semantic adherence to user actions.
Figure 3. Qualitative inverse kinematics comparison. We evaluate how well each method’s predicted representation captures target configurations. Given an object’s rest shape, we optimize its latent coordinates to fit each target and compare the resulting reconstructions. PRIO reproduces the targets most closely.

Limitations and Future Work

  • Topology Constraints: Because the configuration space is continuous and low-dimensional, PRIO cannot currently handle topological changes like tearing, fracturing, or cutting.
  • Material Diversity: The current model is focused on solid deforming objects; it does not yet handle fluids or smoke.
  • Data Scaling: While PRIO generalizes well, the authors note that broadening the pre-training data to include even more diverse physical behaviors is a natural next step for creating truly universal interactive world models.
Figure 5. Interacting with real captured objects. PRIO enables physically driven responses to robot-arm manipulation for objects captured in Deform360 [37]. Ropes bend and plush toys deform naturally without per-object training.

Summary updated

Abstract

3D worlds can now be captured, reconstructed, and generated at scale, yet the objects in them stay still: a pressed pillow does not compress, and a pulled door does not swing. Bringing these objects into interactive 4D world models requires a representation of how they can move, deform, and respond physically to user actions in real time. We present PRIO, a pre-trained representation for interactive objects. Given static geometry, PRIO predicts an object-specific configuration space in a single forward pass. This space provides generalized coordinates for Lagrangian mechanics, allowing physical laws and user actions to govern motion. Configurations are linear in these coordinates, allowing the quantities needed by the dynamics to be precomputed. Dynamics then run at interactive rates without further network inference. Pre-trained on diverse 4D trajectories, PRIO learns priors over object configurations that generalize to unseen shapes. It supports meshes, point clouds, and 3D Gaussians, bringing objects in reconstructed and generated scenes into interaction. We demonstrate its use in real-to-sim workflows and robot manipulation. In addition, compared with existing 4D generation methods, PRIO generates dynamics faster and follows user actions more faithfully.
BibTeX
@misc{huang2026prio,
  author = {Shixun Huang and Lifu Wei and Yue Chang and Chen Geng and Heng Zhang and Jinhyung Ahn and Caoliwen Wang and Siyuan Chen and Donglai Xiang and Jiajun Wu and Peter Yichen Chen},
  title = {{PRIO: Pre-trained Representation for Interactive Objects in 4D Worlds}},
  year = {2026},
  eprint = {2611.25500},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25500},
}
Figure