Chain of Time: In-Context Physical Simulation with Image Generation Models

YingQiao Wang, Eric J. Bigelow, Boyi Li, Tomer D. Ullman

Abstract

Modern Image Generation Models (IGMs) often struggle to maintain physical consistency when predicting future states of a scene. This paper introduces "Chain of Time" (CoT), a zero-shot prompting method that encourages models to simulate physics by generating intermediate frames. While CoT significantly improves accuracy in 2D motion and gravity tasks, it also highlights model limitations in inferring latent physical parameters, such as fluid flow rates, which can lead to compounded errors over time.

Introduction

While Vision-Language Models (VLMs) and IGMs like GPT-4o produce visually impressive results, their internal "world models" frequently fail to adhere to basic physical laws. For instance, predicting the trajectory of a bouncing ball often results in physically impossible outputs. This research aims to improve physical simulation performance without expensive fine-tuning and to provide a method for auditing the internal physical reasoning of these models by forcing them to "show their work."

Overview of the physical simulation tasks and the comparison between baseline and Chain of Time methods.

Methodology

The core contribution is Chain of Time (CoT), a prompting strategy inspired by Chain-of-Thought reasoning and the cognitive theory of mental simulation. Instead of a "Direct Prediction" approach—where a model is given initial frames and asked to jump directly to a distant future state—CoT requires the model to generate a sequence of intermediate frames in-context.

The Chain of Time method involves an iterative loop where generated frames are appended to the context to predict the next step.

As shown in the figure above, the model generates one frame at a time, which is then fed back into the conversation history. This iterative process allows the model to "visualize" the physical progression. To analyze these simulations quantitatively, the researchers used classical computer vision techniques (such as Hough transforms) to extract object coordinates and fluid levels from the generated images, comparing them against ground-truth physical data.

Experimental Setup

The method was evaluated across four distinct physical domains designed to test different aspects of reasoning:

  • 2-D Motion: Constant velocity of a rolling ball on a flat surface.
  • 2-D Gravity: Parabolic trajectories and acceleration of projectile motion.
  • 3-D Fluids: Dynamic filling of a container with water from a pipe.
  • 3-D Bouncing: Real-world video sequences of various balls hitting the ground, testing collisions and elasticity.

Results and Discussion

The "Image Chain-of-Thought" proved highly effective for simple 2D physics. In motion and gravity tasks, CoT significantly outperformed direct prediction; for 2D motion, the Root Mean Square Error (RMSE) dropped from approximately 200 to below 100. However, the study revealed a critical weakness: models are sensitive to salient visual cues (like current water levels) but largely "blind" to latent parameters (like flow rate).

In the Fluids domain, CoT actually performed worse than the baseline. Because the model often misestimated the initial flow rate, generating more intermediate steps simply allowed this error to accumulate, leading to a monotonic increase in error as step size decreased. Similarly, in Bouncing tasks, while the model captured the general trajectory, it consistently underestimated physical elasticity, resulting in "lazy" rebounds compared to reality.

Conclusion

Chain of Time demonstrates that forcing image models to visualize intermediate states can significantly enhance physical reasoning for salient tasks. However, the lack of self-correction and the inability to infer hidden physical constants remain significant hurdles. Future work should focus on enabling models to estimate parameters like mass or friction before simulation and exploring CoT's application to higher-level causal reasoning.

Summary updated

Abstract

We propose a novel method to improve the physical simulation ability of vision-language models. This Chain-of-Time simulation is motivated by in-context reasoning in machine learning, and mental simulation in humans. The method involves generating a series of intermediate images during a simulation. Chain of Time is used at inference time and requires no additional fine-tuning for performance benefits. We apply the Chain-of-Time method to synthetic and real-world domains, including 2-D graphics simulations and natural 3-D videos. These domains test a variety of particular physical properties, including velocity, acceleration, fluid dynamics, and conservation of momentum. We found that using Chain-of-Time simulation substantially improves the performance of state-of-the-art Image Generation Model. Beyond examining performance, we also analyze the specific states of the world simulated by an image model at each time step, which sheds light on the dynamics underlying these simulations. This analysis reveals insights that are hidden from traditional evaluations of physical reasoning, including cases where an Image Generation Model is able to simulate physical properties that unfold over time, such as velocity, gravity, and collisions domain well. Our analysis also highlights particular cases where the Image Generation Model struggles to infer particular physical parameters from input images, despite being capable of simulating relevant physical processes.
BibTeX
@misc{wang2025chain,
  author = {YingQiao Wang and Eric J. Bigelow and Boyi Li and Tomer D. Ullman},
  title = {{Chain of Time: In-Context Physical Simulation with Image Generation Models}},
  year = {2025},
  eprint = {2511.00110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.00110},
}
Figure