PhysResponse: Probing the Physics of Video Generation with Controlled Interventions

Yuhan Li, Shuran Zhang, Haixin Tian, Baonuo Rao, Weipeng Li, Yujia Hou, Zhenyu Wang, Zijun Wang, Luoyang Lin, Lijing Luo, Jiehui Huang, Kaiyuan Yang, Jingbin Tan, Panwen Hu, Xiaodan LiangarXiv 2026 · arXiv

TL;DR

PhysResponse is a new benchmark and dataset designed to move beyond "looking real" to "acting right." It evaluates whether video generation models actually respond to specific physical parameters like gravity, friction, and speed, revealing that current state-of-the-art models fail to accurately simulate these dynamics even when prompted with exact numerical values.

What Problem Does This Paper Address?

In the race to build "world simulators," researchers have focused heavily on physical plausibility—whether a generated video looks like it follows the laws of physics. However, the authors argue that a true simulator must also exhibit physical parameter responsiveness.

If you tell a model that gravity is as weak as it is on the Moon ($1.62\text{ m/s}^2$), the objects should fall slowly. If you increase the friction of a surface, a sliding block should stop sooner. Current models often ignore these numerical constraints, producing "plausible" motion that doesn't actually match the user's prompt. This gap limits the use of video models in fields like robotics (Embodied AI) or scientific discovery, where precise control over the environment is mandatory.

Methods

The authors introduce a two-level evaluation framework:

  1. Scene-Conditioned Simulator Inversion: To measure the "effective" physics of a generated video, they use a physics engine (Genesis or Isaac Sim) to find the physical parameters that best match the generated motion. If the model was prompted for Earth gravity but the objects fall like they are on Jupiter, the "inversion" process identifies that discrepancy.
  2. Event Evaluation: They test models on "critical transitions"—points where a tiny change in a parameter changes the outcome. For example, a book pushed at $0.47\text{ m/s}$ might stay on a table, but at $0.73\text{ m/s}$ it might tip a can over. They measure if models can correctly reproduce these "outcome flips."
Figure 2: Speed decides the outcome in simulation but often not in generation.

To improve model performance, they also experiment with a numerical encoder that converts physical values into sinusoidal embeddings, helping the model "understand" numbers better than raw text tokens.

Datasets

The paper introduces the PhysResponse Dataset, which consists of:

  • 72,000 simulated videos covering 20 multi-object interaction types (e.g., dominoes, ramps, multi-stage cascades).
  • Controlled Parameters: Each video has precisely set gravity, friction, and initial speeds.
  • Core100 Benchmark: A curated subset of 100 cases used to evaluate ten leading image-to-video models (including Wan, Cosmos, and CogVideo).
Figure 3: The pipeline for constructing the PhysResponse dataset.

Key Insights and Ideas

  • Plausibility is a Trap: A model can achieve high visual scores (like PSNR or SSIM) by simply repeating the first frame or generating slow, blurry motion. However, these models fail miserably at physics. For instance, some models "cheat" by showing an object that barely moves, which looks "plausible" but ignores the prompted initial speed.
  • Massive Parameter Errors: Even the best-performing models have parameter errors (gravity, friction, speed) that are an order of magnitude higher than ground-truth simulations.
  • Complementary Failures: The authors found that a model might get the event right (e.g., the can falls) but the dynamics wrong (e.g., it falls at the wrong speed), or vice versa. This highlights why both levels of evaluation are necessary.
  • Fine-tuning Transfers: Training models on the synthetic PhysResponse dataset significantly improved their performance on real-world video benchmarks and held-out interaction types, suggesting that "learning physics" from simulation is a viable path for general video models.

Key Results

  • Baseline Failure: Ten SOTA models were tested; nine of them scored between 1% and 28% on causal order accuracy (compared to 99% for ground truth).
  • Gravity Mismatch: In one striking example, a model prompted with Moon gravity ($1.62\text{ m/s}^2$) generated motion that actually matched a gravity of $11.8\text{ m/s}^2$.
Figure 6: Recovering gravity by matching generated motion to physics simulations.
  • Fine-tuning Gains: Fine-tuning a baseline model (Wan2.2-TI2V-5B) on PhysResponse reduced gravity errors from $94.9\%$ to $69.1\%$ and improved "human measurability" (how often the video actually shows clear motion) from $35\%$ to $80\%$.

Limitations and Future Work

  • Conditional Identifiability: In some scenes, it is hard to distinguish between the effects of gravity and friction (e.g., a sliding block). The authors suggest that more complex, multi-stage interactions are needed to fully "disentangle" these parameters.
  • Scope of Interactions: The current dataset focuses on rigid-body physics. Future work could expand to fluids, soft bodies, or complex human-object interactions.
  • Embodied AI: While the paper provides a foundation for world simulators, the authors haven't yet tested if these improved models actually help robots learn better policies in the real world.

Summary updated

Abstract

Video generation models are increasingly viewed as world simulators, which requires not only plausible motion but dynamics that respond correctly to specified physical parameters. Existing models, datasets and benchmarks, however, largely target physical plausibility, leaving this physical parameter responsiveness insufficiently explored. We introduce PhysResponse, a benchmark that measures it at two levels: scene-conditioned simulator inversion recovers the effective gravity, friction and initial speed of generated motion, and event evaluation tests outcomes near and across critical transitions. Evaluating ten image-to-video models, we find that even the lowest parameter errors exceed those of physically correct videos by over an order of magnitude, that models rarely reproduce both outcomes of a transition pair, and that the two levels expose complementary failures that visual metrics such as PSNR miss. To supply the missing training signal, we construct 72,000 simulated videos of 20 multi-object interaction types with controlled parameters; fine-tuning on them lowers parameter errors, even on held-out interaction types, and transfers to real videos and an external physics benchmark. Continued training with a numerical encoder further lowers all three errors on held-out types. PhysResponse provides a foundation for world simulators that follow physical conditions, not just appearances.
BibTeX
@misc{li2026physresponse,
  author = {Yuhan Li and Shuran Zhang and Haixin Tian and Baonuo Rao and Weipeng Li and Yujia Hou and Zhenyu Wang and Zijun Wang and Luoyang Lin and Lijing Luo and Jiehui Huang and Kaiyuan Yang and Jingbin Tan and Panwen Hu and Xiaodan Liang},
  title = {{PhysResponse: Probing the Physics of Video Generation with Controlled Interventions}},
  year = {2026},
  eprint = {2611.24419},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24419},
}
Figure