PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation

Introduction

PhysCtrl is a novel framework that brings physical "common sense" to video generation by learning the underlying laws of motion. By predicting 3D point trajectories conditioned on material properties (like stiffness) and external forces, it allows users to generate videos that are not only photorealistic but also physically accurate and highly controllable.

The Challenge of "Physics Hallucinations"

While modern video generation models like Sora or Gen-3 produce stunning visuals, they often suffer from "physics hallucinations." Objects might morph unnaturally, gravity seems optional, and materials lack consistent properties—a rubber ball might behave like a rock in one frame and a liquid in the next.

The core challenge is that these models are trained on 2D pixels without an explicit understanding of the 3D physical world. Furthermore, existing "controllable" models usually rely on simple 2D drag-and-drop interfaces that don't account for depth or material resistance. PhysCtrl addresses this gap by bridging the world of traditional physics simulators (which are accurate but slow and hard to tune) with generative diffusion models (which are fast and expressive).

How PhysCtrl Works

The PhysCtrl pipeline consists of three main stages designed to ground generation in physical reality:

1. Point Cloud Lifting: The system takes a single 2D image and "lifts" the subject into a 3D point cloud representation.
2. Generative Physics Network: The heart of the framework is a diffusion-based transformer that predicts how those 3D points will move over time. Crucially, it is conditioned on:
3. Material Type: Categories include elastic, sand, plasticine, or rigid.
4. Physical Parameters: Properties such as Young’s Modulus ($E$), which defines stiffness.
5. External Forces: User-defined "pulls" or "pushes" applied to specific points to direct motion.

Expand

Source, p. 1

1. Video Synthesis: The generated 3D trajectories are projected back into 2D and used as a "control signal" to drive a pre-trained image-to-video model, ensuring the final pixels follow the predicted physical motion.

Technical Innovations

• Spatiotemporal Attention: Instead of naive 4D attention, the authors use a block that first performs spatial attention (allowing particles to "talk" to their neighbors, mimicking real-world particle dynamics) followed by temporal attention.
• Physics-Grounded Constraints: During training, the model isn't just told to match the ground truth; it is penalized if its predicted motion violates the Material Point Method (MPM) equations, effectively "baking" the laws of physics into the neural network.

Training on the "Rules of the Universe"

To teach the model these physical laws, the authors created a massive synthetic dataset featuring 550K animations generated using high-fidelity physics simulators (MPM and rigid-body solvers). The dataset covers four distinct material categories and sources geometry from the ObjaverseXL dataset to ensure the model generalizes to a wide variety of shapes.

Key Insights and Capabilities

Trajectories as the Universal Language By representing motion as 3D point trajectories rather than 2D optical flow or latent features, the model gains a representation that is naturally compatible with both physics simulators and 3D-aware video models.

Learning vs. Simulating Traditional simulators require precise, manual hyperparameter tuning and are computationally expensive. PhysCtrl demonstrates that a generative model can learn the distribution of these simulations, allowing for near-instant "physics-plausible" inference.

Material-Aware Control A major breakthrough is the ability to change a single parameter (like Young's Modulus) and see the same object react differently to the same force. An object might bend slightly if it's "stiff" or deform significantly if it's "soft."

Expand

Source, p. 1

Results and Performance

PhysCtrl significantly outperforms existing state-of-the-art controllable video models:

• Trajectory Accuracy: In terms of volume Intersection over Union (vIoU), PhysCtrl achieved 77.59%, vastly outperforming the previous best of 53.78% (MDM).
• Human/LLM Evaluation: Using GPT-4o and user studies to rank "Physical Plausibility," PhysCtrl was preferred over baselines like CogVideoX and DragAnything by a wide margin (81% preference for physics plausibility in user studies).
• Efficiency: While a traditional differentiable simulator might take 3 minutes to estimate physics parameters for an inverse problem, PhysCtrl’s learned prior can do it in about 0.1 seconds.

Limitations and Future Work

While groundbreaking, the authors acknowledge several areas for improvement:

• Thin Structures: Like many point-cloud-based methods, the model can struggle with very thin geometries.
• Complex Internal Deformations: While it handles surface motion well, highly complex internal "squishing" of volumes isn't always perfectly captured.
• Multi-Object Interaction: The current model focuses primarily on single-object dynamics. Scaling this to complex, multi-object scenes remains a future challenge.
• Material Breadth: The model is currently limited to four material types and does not yet handle fluids or gases.

@inproceedings{wang2025physctrl,
  author = {Chen Wang and Chuhao Chen and Yiming Huang and Zhiyang Dou and Yuan Liu and Jiatao Gu and Lingjie Liu},
  title = {{PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=AHEKhff4Oa},
}