Chain of Time: In-Context Physical Simulation with Image Generation Models

YingQiao Wang, Eric Bigelow, Boyi Li, Tomer UllmanSubmission to ICLR 2026 · Rejected

TL;DR

Researchers have introduced Chain of Time, a prompting method that enables image generation models to simulate physical processes by generating a sequence of intermediate frames. Inspired by human mental simulation, this "step-by-step" approach significantly improves prediction accuracy in most physical domains and provides a rare window into the internal "physics engines" of models like GPT-4o.

What Problem Does This Paper Address?

While modern Vision-Language Models (VLMs) and Image Generation Models (IGMs) can produce stunningly realistic images, they often stumble when asked to predict how a scene will unfold over time. They might "teleport" an object to a final destination or ignore basic laws like gravity and momentum.

Existing research usually treats these models as black boxes—measuring if they get the final answer right, but not how they got there. The authors identify two main challenges:

  1. The Prediction Gap: Direct prediction of a future state (e.g., "where will this ball be in 1 second?") is a high-leap task for models.
  2. Interpretability: We lack tools to see the incremental "reasoning" steps a model takes when simulating physics.

Methods

The authors propose Chain of Time (CoT-time), an inference-time prompting technique that requires no additional fine-tuning.

  • The Concept: Much like "Chain of Thought" prompting forces an LLM to show its work through text, Chain of Time forces an IGM to show its work through images. Instead of jumping from an initial state ($I_t$) to a distant future state ($I_T$), the model is prompted to generate a series of intermediate frames at small time steps ($k$).
  • The Workflow:
    1. Initial Prompt: The model receives a sequence of starting frames and is asked to generate the next frame $k$ seconds into the future.
  • Iterative Follow-up: The model's own generated output is fed back in as context to generate the subsequent frame, creating a linearly unfolding Markov Chain of images.
  • The "Auditor": To evaluate the results, the researchers used classic computer vision algorithms (like Hough transforms and color thresholding) to extract coordinates and water levels from the generated images, allowing for a quantitative comparison against ground-truth physics.

Datasets

The study tested the method across four distinct physical domains:

  • 2D Motion: A red ball rolling on a flat surface (testing velocity).
  • 2D Gravity: Projectile motion where a ball follows a parabolic arc (testing gravity).
  • 3D Fluids: A pipe filling a glass mug with water (testing flow rate and fluid dynamics).
  • 3D Bouncing: Real-world-style videos of different balls (tennis, squash, soccer) hitting the ground (testing the "coefficient of restitution" or bounciness).

Key Insights and Ideas

  • Process vs. Parameter: One of the most striking findings is that IGMs are often capable of simulating the qualitative process (e.g., they know water should rise or a ball should bounce up) but fail at the quantitative parameters (e.g., they don't accurately perceive how fast the water is flowing or how "bouncy" a specific ball is).
  • Auditing the Internal World Model: By forcing the model to generate intermediate steps, the researchers could see exactly where the simulation diverged from reality. For example, in the bouncing task, the model correctly simulated the "U-shaped" trajectory but consistently predicted the ball would bounce back slower than it actually does.
  • The "Fluids" Paradox: In the Fluids domain, more simulation steps actually increased error. The analysis revealed that the model was sensitive to the initial water level (the "intercept") but almost completely ignored the flow rate (the "slope"). Because the model's initial estimate of the flow rate was wrong, generating more frames simply allowed that error to compound over time.

Key Results

  • Significant Accuracy Gains: In the 2D Motion task, Chain of Time (with 0.2s steps) more than halved the error compared to direct prediction.
  • Monotonic Improvement: In 3 out of 4 domains, the finer the time step (the more "links" in the chain), the more accurate the final prediction became.
  • GPT-4o as a Simulator: The study found that while models like DALL-E 3 struggled to simulate future states with any accuracy, GPT-4o (as of late 2024/early 2025) demonstrated a nascent but functional internal physics engine.

Limitations and Future Work

  • Computational Cost: Generating a sequence of images is significantly more expensive and time-consuming than generating a single "direct prediction" image.
  • Initial Perception Errors: The method is highly dependent on the model's ability to correctly "read" physical parameters from the starting frames. If the model misjudges the weight or speed of an object at the start, the Chain of Time will accurately simulate a wrong physical reality.
  • Future Directions: The authors suggest exploring how models might be prompted to reason about causality (e.g., "what caused this ball to stop?") or using non-simulation heuristics to supplement the step-by-step generation.

Summary updated

Abstract

We propose a novel method to improve the physical simulation ability of vision-language models. This Chain-of-Time simulation is motivated by in-context reasoning in machine learning, and mental simulation in humans. The method involves generating a series of intermediate images during a simulation. Chain of Time is used at inference time and requires no additional fine-tuning for performance benefits. We apply the Chain-of-Time method to synthetic and real-world domains, including 2-D graphics simulations and natural 3-D videos. These domains test a variety of particular physical properties, including velocity, acceleration, fluid dynamics, and conservation of momentum. We found that using Chain-of-Time simulation substantially improves the performance of state-of-the-art Image Generation Model. Beyond examining performance, we also analyze the specific states of the world simulated by an image model at each time step, which sheds light on the dynamics underlying these simulations. This analysis reveals insights that are hidden from traditional evaluations of physical reasoning, including cases where an Image Generation Model is able to simulate physical properties that unfold over time, such as velocity, gravity, and collisions domain well. Our analysis also highlights particular cases where the Image Generation Model struggles to infer particular physical parameters from input images, despite being capable of simulating relevant physical processes.
BibTeX
@misc{wang2025chain,
  author = {YingQiao Wang and Eric Bigelow and Boyi Li and Tomer Ullman},
  title = {{Chain of Time: In-Context Physical Simulation with Image Generation Models}},
  year = {2025},
  eprint = {2511.00110},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.00110},
}
Figure