TL;DR
PhysResponse is a new benchmark and dataset designed to move beyond "looking real" to "acting right." It evaluates whether video generation models actually respond to specific physical parameters like gravity, friction, and speed, revealing that current state-of-the-art models fail to accurately simulate these dynamics even when prompted with exact numerical values.
What Problem Does This Paper Address?
In the race to build "world simulators," researchers have focused heavily on physical plausibility—whether a generated video looks like it follows the laws of physics. However, the authors argue that a true simulator must also exhibit physical parameter responsiveness.
If you tell a model that gravity is as weak as it is on the Moon ($1.62\text{ m/s}^2$), the objects should fall slowly. If you increase the friction of a surface, a sliding block should stop sooner. Current models often ignore these numerical constraints, producing "plausible" motion that doesn't actually match the user's prompt. This gap limits the use of video models in fields like robotics (Embodied AI) or scientific discovery, where precise control over the environment is mandatory.
Methods
The authors introduce a two-level evaluation framework:
- Scene-Conditioned Simulator Inversion: To measure the "effective" physics of a generated video, they use a physics engine (Genesis or Isaac Sim) to find the physical parameters that best match the generated motion. If the model was prompted for Earth gravity but the objects fall like they are on Jupiter, the "inversion" process identifies that discrepancy.
- Event Evaluation: They test models on "critical transitions"—points where a tiny change in a parameter changes the outcome. For example, a book pushed at $0.47\text{ m/s}$ might stay on a table, but at $0.73\text{ m/s}$ it might tip a can over. They measure if models can correctly reproduce these "outcome flips."
To improve model performance, they also experiment with a numerical encoder that converts physical values into sinusoidal embeddings, helping the model "understand" numbers better than raw text tokens.
Datasets
The paper introduces the PhysResponse Dataset, which consists of:
- 72,000 simulated videos covering 20 multi-object interaction types (e.g., dominoes, ramps, multi-stage cascades).
- Controlled Parameters: Each video has precisely set gravity, friction, and initial speeds.
- Core100 Benchmark: A curated subset of 100 cases used to evaluate ten leading image-to-video models (including Wan, Cosmos, and CogVideo).
Key Insights and Ideas
- Plausibility is a Trap: A model can achieve high visual scores (like PSNR or SSIM) by simply repeating the first frame or generating slow, blurry motion. However, these models fail miserably at physics. For instance, some models "cheat" by showing an object that barely moves, which looks "plausible" but ignores the prompted initial speed.
- Massive Parameter Errors: Even the best-performing models have parameter errors (gravity, friction, speed) that are an order of magnitude higher than ground-truth simulations.
- Complementary Failures: The authors found that a model might get the event right (e.g., the can falls) but the dynamics wrong (e.g., it falls at the wrong speed), or vice versa. This highlights why both levels of evaluation are necessary.
- Fine-tuning Transfers: Training models on the synthetic PhysResponse dataset significantly improved their performance on real-world video benchmarks and held-out interaction types, suggesting that "learning physics" from simulation is a viable path for general video models.
Key Results
- Baseline Failure: Ten SOTA models were tested; nine of them scored between 1% and 28% on causal order accuracy (compared to 99% for ground truth).
- Gravity Mismatch: In one striking example, a model prompted with Moon gravity ($1.62\text{ m/s}^2$) generated motion that actually matched a gravity of $11.8\text{ m/s}^2$.
- Fine-tuning Gains: Fine-tuning a baseline model (Wan2.2-TI2V-5B) on PhysResponse reduced gravity errors from $94.9\%$ to $69.1\%$ and improved "human measurability" (how often the video actually shows clear motion) from $35\%$ to $80\%$.
Limitations and Future Work
- Conditional Identifiability: In some scenes, it is hard to distinguish between the effects of gravity and friction (e.g., a sliding block). The authors suggest that more complex, multi-stage interactions are needed to fully "disentangle" these parameters.
- Scope of Interactions: The current dataset focuses on rigid-body physics. Future work could expand to fluids, soft bodies, or complex human-object interactions.
- Embodied AI: While the paper provides a foundation for world simulators, the authors haven't yet tested if these improved models actually help robots learn better policies in the real world.
Summary updated