PhysicsGen: Can Generative Models Learn from Images to Predict Complex Physical Relations?

Martin Spitznagel, Jan Vaillant, Janis Keuper

TL;DR

This paper introduces PhysicsGen, a novel benchmark for evaluating generative models' ability to learn complex physical relationships from image pairs. While current models show promise in speedups, they struggle with higher-order physical phenomena, highlighting the need for new methods to enforce physical correctness.

What Problem Does This Paper Address?

The paper tackles the challenge of whether generative AI models, particularly those excelling at image-to-image translation, can learn and accurately predict complex physical phenomena solely from visual input. Traditionally, simulating physical processes relies on solving differential equations, which can be computationally expensive. The authors explore if generative models can offer a faster, data-driven alternative for these simulations, bridging the gap between visual understanding and physical accuracy. They aim to answer: can these models learn intricate physical laws, and what speedups can be achieved by replacing traditional solvers?

Methods

The core of the paper is the introduction of the PhysicsGen benchmark. This benchmark comprises three distinct physical simulation tasks:

  1. Urban Sound Propagation: Simulating how sound waves travel and interact in urban environments, involving diffraction and reflections.
  2. Lens Distortion Correction: Modeling and correcting optical distortions in images, specifically using the Brown-Conrady model.
  3. Dynamics of Rolling and Bouncing Balls: Simulating the motion of balls on inclined surfaces, incorporating linear and rotational dynamics.

For evaluation, the authors employ standard image-to-image generative models, including GANs (Pix2Pix), U-Nets, Diffusion Models (DDPM, Stable Diffusion), and Autoencoders (VAE, ConvAE). They measure performance based on both speedups (runtime) and physical correctness (accuracy metrics tailored to each task).

Datasets

The PhysicsGen benchmark introduces a new dataset consisting of 300,000 image-pairs across the three physical simulation tasks:

  • Urban Sound Propagation: 100,000 image-pairs derived from urban layouts and simulated sound propagation maps.
  • Lens Distortion Correction: 100,000 image-pairs from the CelebA dataset, with distortions applied using the Brown-Conrady model.
  • Rolling and Bouncing Ball Dynamics: 100,000 image-pairs simulating the motion of balls on inclined surfaces.

Key Insights and Ideas

  • Generative models can achieve significant speedups: Replacing traditional differential equation solvers with generative models can lead to substantial computational efficiency gains, especially for complex iterative simulations.
  • Current models struggle with higher-order physics: While generative models perform well on simpler (0th and 1st order) physical phenomena, they exhibit significant limitations when dealing with more complex relationships that involve higher-order derivatives or intricate interactions.
  • Distinct error patterns emerge across model types: Different architectures (GANs, U-Nets, Diffusion models) exhibit unique failure modes and error characteristics, offering insights into their respective strengths and weaknesses in capturing physical realism.
  • Physical correctness remains a key challenge: Despite impressive visual outputs, models often fail to fully capture the underlying physical laws, leading to plausible-looking but physically inaccurate predictions.

Key Results

  • Speedups: Generative models can achieve speedups of up to 20,000x compared to traditional simulation methods, particularly for iterative tasks like urban sound propagation.
  • Sound Propagation: Models show good results for simpler scenarios but struggle with reflections and diffractions, leading to increased errors in non-line-of-sight regions. Pix2Pix and UNet were noted for their performance.
  • Lens Distortion: Generative models can replicate distortions, but accuracy in predicting facial landmarks is task-dependent. Errors increase with the complexity of distortion coefficients.
  • Ball Dynamics: Pix2Pix and UNet generally performed best in simulating rolling and bouncing balls. Diffusion models showed promise but often produced blurred outputs or struggled with higher-order dynamics. Autoencoders performed poorly, with a high percentage of non-evaluable predictions.

Limitations and Future Work

  • Computational Cost of Simulations: The authors note that the runtime analysis of simulations was not fully optimized for GPUs, potentially affecting direct comparisons.
  • Generalizability: While the chosen tasks are diverse, they represent only a subset of possible physical simulations, and results may not transfer to all scenarios.
  • Need for Physics-Informed Models: The observed limitations in predicting higher-order physical relations suggest a need for new methods to explicitly enforce physical correctness, possibly through physics-informed training or architectural modifications.
  • Data Limitations: The paper raises the question of whether current limitations are due to inherent model constraints or insufficient training data, suggesting further investigation into data scaling.

Summary updated

BibTeX
@inproceedings{spitznagel2025physicsgen,
  author = {Martin Spitznagel and Jan Vaillant and Janis Keuper},
  title = {{PhysicsGen: Can Generative Models Learn from Images to Predict Complex Physical Relations?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01039},
}

Loading related content...

Figure