TL;DR
PRIO is a pre-trained model that transforms static 3D objects—whether meshes, point clouds, or 3D Gaussians—into interactive, physically responsive "4D" entities in a single forward pass. By predicting a linear configuration space, it enables real-time physics simulations that respond to user actions at over 30 FPS without requiring per-object training or expensive neural inference at every time step.
What Problem Does This Paper Address?
We are currently in an era where 3D worlds can be captured or generated at scale, but these worlds are largely "frozen." A reconstructed pillow doesn't compress when pressed, and a generated door doesn't swing when pulled.
Existing solutions to this "static world" problem fall into two camps, both with significant drawbacks:
- Generative Video Models: These produce convincing motion but output pixels rather than consistent 3D objects, and they require heavy computation for every single frame.
- Physics-Based Simulation: Traditional methods can be physically accurate but often take minutes or hours to compute. Recent "learned" simulators either require training a new model for every specific object or use non-linear spaces that require running a neural network at every step of the simulation, killing real-time performance.
PRIO bridges this gap by providing a representation that is both generalizable (works on unseen objects) and efficient (runs at interactive rates).
Methods
The core innovation of PRIO is predicting a linear configuration space for an object based solely on its static geometry.
The Architecture
- Shape Encoding: PRIO samples points from the object and uses a DiffusionNet to extract local and global geometric features.
- Shape Tokens: These features are condensed into a set of "shape tokens" via self-attention, summarizing different regions of the object.
- Field Prediction: A decoder uses cross-attention between query points and these tokens to predict a continuous field, $\Phi_G(X)$. This field defines the "basis" of how the object can move and deform.
- Linearity: Because the deformation is modeled as a linear combination of these basis fields ($x = X + \Phi q$), the heavy lifting of the physics (the mass matrices and potential derivatives) can be precomputed once.
The Physics
Once the linear space is predicted, PRIO uses Lagrangian dynamics. User actions and physical laws (like gravity or elasticity) drive a small set of generalized coordinates ($q$). Because the mapping is linear, the simulation becomes a series of efficient matrix-vector multiplications that run entirely on the GPU without further network inference.
Datasets
- Pre-training: PRIO is pre-trained on a diverse collection of simulated 4D trajectories (deforming meshes and objects).
- Evaluation:
- Deform360: A dataset of real-world objects deforming under user interaction, used to test zero-shot transfer to real captures.
- Unseen Shapes: Various 3D models not present in the training set to test generalization.
- Benchmarks: Compared against specialized rigging and skinning methods like PhysSkin, UniRig, and NeuROK.
Key Insights and Ideas
- Linearity is a Feature, Not a Bug: While non-linear spaces can be more expressive, they are computationally expensive. PRIO shows that a well-predicted linear space is sufficient for high-quality, real-time interaction.
- Encoder-Free Training: Instead of training an encoder to find latent coordinates, PRIO solves for the optimal coordinates during training using a closed-form least-squares solution (the pseudo-inverse). This ensures the model learns the best possible configuration space the decoder can represent.
- Format Agnostic: Because PRIO predicts a continuous field, it doesn't care if your input is a mesh, a point cloud, or 3D Gaussians. It can drive the "interactive 4D" version of any of them.
Key Results
- Superior Accuracy: In inverse kinematics tasks (fitting the model to a target shape), PRIO outperformed the next best method (PhysSkin) by a significant margin, reducing Chamfer $L_1$ error from 0.01299 to 0.00522.
- Speed: PRIO runs 7.1x faster than Vertex Block Descent (VBD) for similar accuracy levels.
- Real-Time Performance: The system consistently runs at interactive rates. For example, it can simulate a scene with eight plush toys (nearly 1M triangles) at 25ms per step on a single GPU.
- Physical Plausibility: Using Vision-Language Model (VLM) scoring, PRIO was rated significantly higher than generative baselines for physical common sense and semantic adherence to user actions.
Limitations and Future Work
- Topology Constraints: Because the configuration space is continuous and low-dimensional, PRIO cannot currently handle topological changes like tearing, fracturing, or cutting.
- Material Diversity: The current model is focused on solid deforming objects; it does not yet handle fluids or smoke.
- Data Scaling: While PRIO generalizes well, the authors note that broadening the pre-training data to include even more diverse physical behaviors is a natural next step for creating truly universal interactive world models.
Summary updated