Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language

Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Joshua B. Tenenbaum, Chuang Gan

Imagine watching a video of a billiard break. As a human, you don’t just see moving pixels; you intuitively understand that the cue ball has momentum, the table has friction, and if the red ball were twice as heavy, it wouldn't have traveled as far. Most current AI models struggle with this "intuitive physics," often treating video reasoning as a pattern-matching exercise rather than a physical simulation.

In their paper, "Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language," researchers from MIT, HKU, and IBM introduce VRDP (Visual Reasoning with Differentiable Physics). This framework allows AI to jointly learn visual concepts and infer the underlying physical properties of objects simply by watching videos and reading related questions.

TL;DR

VRDP is a neuro-symbolic framework that integrates a differentiable physics engine into a visual reasoning pipeline. By "fitting" a physical simulation to observed video trajectories, it can answer complex "what-if" (counterfactual) and predictive questions with state-of-the-art accuracy and remarkable data efficiency.

What Problem Does This Paper Address?

Dynamic visual reasoning—explaining what happened, predicting what will happen, and imagining counterfactuals—is a cornerstone of human intelligence. However, existing AI approaches generally fall into two camps, both with flaws:

  1. Pure Neural Networks: These "black box" models are great at pattern recognition but lack transparency and often fail at long-term predictions because they don't "understand" the laws of physics.
  2. Standard Neuro-Symbolic Models: While more interpretable, these models usually don't explicitly incorporate physical laws (like conservation of momentum), making them struggle when asked to imagine scenarios that haven't happened yet.

The challenge is creating a model that combines the perception of deep learning with the rigorous, predictable constraints of a physics engine.

Methods

The VRDP framework consists of three seamlessly integrated modules:

The VRDP framework architecture
  1. Visual Perception Module: Uses a pre-trained Faster R-CNN to detect objects and track their trajectories across video frames, representing them as latent scene representations.
  2. Concept Learner: This module grounds linguistic concepts (like "cyan," "cube," or "collision") into visual features. It translates questions into executable symbolic programs.
  3. Differentiable Physics Engine: This is the core innovation. It’s an impulse-based rigid-body simulator. Because it is differentiable, the model can use gradient-based optimization (specifically L-BFGS) to "tweak" physical parameters—such as mass, restitution (bounciness), and friction—until the simulated trajectory perfectly matches the observed video.

Once the physics are "solved" for a video, the model can rerun the simulation with modified parameters (e.g., "What if the ball was heavier?") to answer counterfactual questions.

Datasets

The authors evaluated VRDP on three primary benchmarks:

  • CLEVRER: A standard synthetic dataset for causal reasoning in videos involving colliding geometric shapes.
  • Real-Billiard: A real-world dataset of three-cushion billiard games, testing the model's ability to handle noisy, non-synthetic dynamics.
  • Generalized CLEVRER: A newly introduced few-shot dataset designed to test how quickly the model can learn novel concepts like "heavier" or "lighter" with minimal training data (only 25 videos).

Key Insights and Ideas

  • Physics as a Strong Prior: Instead of forcing a neural network to learn the concept of "gravity" or "momentum" from scratch, VRDP uses the actual equations of physics as a constraint. This makes the model's predictions physically grounded by design.
  • Analysis-by-Synthesis: The model learns by trying to recreate the video it just saw. If the simulation doesn't match the video, it knows its estimate of the object's mass or velocity is wrong and adjusts it.
  • Extreme Data Efficiency: Because the model understands the "rules of the game" (physics), it doesn't need millions of examples. It can often infer the physical properties of a scene from a single video.
  • Transparency: Unlike black-box models, you can inspect VRDP’s "thoughts." You can see the exact mass and friction values it calculated, making the reasoning process fully interpretable.

Key Results

Performance comparison showing VRDP's data efficiency
  • State-of-the-Art Accuracy: On the CLEVRER benchmark, VRDP improved the accuracy of predictive and counterfactual questions by 4.5% and 11.5% respectively over the previous best models.
  • Superior Efficiency: As shown in the performance charts, VRDP achieved comparable results to top-tier models while using up to 3x less training data. In the few-shot "Generalized CLEVRER" task, it outperformed competitors by a massive margin (75.6% vs 26.7% accuracy).
  • Real-World Performance: On the Real-Billiard dataset, VRDP significantly reduced prediction error compared to pure neural approaches, proving that differentiable physics can bridge the gap from simulation to reality.

Limitations and Future Work

Despite its success, the authors note several areas for improvement:

  • Collision Dependency: The model struggles to estimate mass or restitution if an object never collides with anything in the video.
  • Initial Conditions: If an object is involved in a collision the very moment it enters the frame, the model lacks enough "clean" data to estimate its initial velocity.
  • Geometric Complexity: The current engine handles spheres and cylinders well, but complex shapes (like cubes) are more difficult to simulate accurately due to the intricacies of corner-based collisions.

Future work may involve extending the engine to handle soft-body dynamics, fluids, or more complex multi-object interactions.

Summary updated

Abstract

In this work, we propose a unified framework, called Visual Reasoning with Differ-entiable Physics (VRDP), that can jointly learn visual concepts and infer physics models of objects and their interactions from videos and language. This is achieved by seamlessly integrating three components: a visual perception module, a concept learner, and a differentiable physics engine. The visual perception module parses each video frame into object-centric trajectories and represents them as latent scene representations. The concept learner grounds visual concepts (e.g., color, shape, and material) from these object-centric representations based on the language, thus providing prior knowledge for the physics engine. The differentiable physics model, implemented as an impulse-based differentiable rigid-body simulator, performs differentiable physical simulation based on the grounded concepts to infer physical properties, such as mass, restitution, and velocity, by fitting the simulated trajectories into the video observations. Consequently, these learned concepts and physical models can explain what we have seen and imagine what is about to happen in future and counterfactual scenarios. Integrating differentiable physics into the dynamic reasoning framework offers several appealing benefits. More accurate dynamics prediction in learned physics models enables state-of-the-art performance on both synthetic and real-world benchmarks while still maintaining high transparency and interpretability; most notably, VRDP improves the accuracy of predictive and counterfactual questions by 4.5% and 11.5% compared to its best counterpart. VRDP is also highly data-efficient: physical parameters can be optimized from very few videos, and even a single video can be sufficient. Finally, with all physical parameters inferred, VRDP can quickly learn new concepts from a few examples.
BibTeX
@inproceedings{ding2021dynamic,
  author = {Mingyu Ding and Zhenfang Chen and Tao Du and Ping Luo and Joshua B. Tenenbaum and Chuang Gan},
  title = {{Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=lk1ORT35tbi},
}
Figure