Interpreting Physics in Video World Models

Sonia Joseph, Quentin Garrido, Randall Balestriero, Matthew Kowal, Thomas Fel, Shahab Bakhtiari, Blake Richards, Mike Rabbat

TL;DR

By peering inside state-of-the-art video world models like V-JEPA 2 and VideoMAE-v2, researchers have discovered that physical understanding emerges suddenly at exactly one-third of the network's depth—a region dubbed the Physics Emergence Zone. Rather than building a classical, clean "physics engine" with reusable variables, these models rely on highly distributed, task-specific, and brain-like representations to predict the physical world.

What Problem Does This Paper Address?

Modern video world models achieve remarkable performance on intuitive physics benchmarks, yet they are typically treated as black boxes. Within cognitive science, a long-standing debate questions whether physical reasoning requires an explicit "intuitive physics engine" (which uses compact, factorized variables like velocity, mass, and direction) or if it can be achieved via distributed, heuristic shortcuts.

This paper presents the first comprehensive interpretability study to directly map how large-scale video transformers internally construct, organize, and geometrically represent physical variables.

Methods

The authors employ a diverse toolkit of mechanistic interpretability techniques to dissect the frozen encoders of video transformers:

  • Layerwise Probing: Training linear and attentive-MLP probes on the model's residual stream at every layer to determine where specific physical properties become readable.
  • Subspace Overlap Analysis: Measuring principal angles and projection overlaps between different task subspaces to evaluate whether distinct physical tasks share the same underlying latent features.
  • Targeted Attention Ablations: Masking out spatiotemporally local attention weights within specific layers to establish their causal role in processing physical dynamics.
  • Activation Steering (Orthogonal Probe Sequence Method): Iteratively training linear probes, projecting out their directions, and retraining to determine the exact dimensionality of a concept and attempt to causally manipulate it.

Datasets

The study evaluates models across several specialized and control datasets:

  • IntPhys: A benchmark containing pairs of physically possible vs. impossible videos (e.g., objects spontaneously disappearing or violating trajectory continuity) that differ only at a single "breakpoint" frame.
  • Synthetic Toy Ball Dataset: A newly introduced dataset generated via the Kubric simulator featuring a single sphere moving with precisely controlled straight-line trajectories, constant velocities, or uniform accelerations to provide clean ground-truth physical variables.
  • CLEVRER: A multi-object video dataset used to validate whether the internal representations generalize to more complex scenes.
  • Control Datasets: ImageNet (static image classification) and Something-Something-v2 (action recognition) to verify task specificity.

Key Insights and Ideas

  • The Physics Emergence Zone (PEZ): Across different model architectures and scales, physical variables are virtually unreadable in early layers but become sharply accessible at roughly one-third of the network's depth.
  • The Middle-Layer Peak: Physical representations peak in accuracy within the middle third of the network and actually degrade toward the final output layers. The final layers discard explicit physical variables to optimize for the model's specific training objective (e.g., predicting missing patches in latent space).
  • No Shared Physics Engine: If these models operated like a classical physics engine, a variable like "motion direction" would be compositionally reused to judge if a video is physically possible. Instead, the representational subspaces for motion direction and possible/impossible physics judgments are nearly orthogonal, indicating highly task-specific representations.
  • Asymmetric Motion Processing: Scalar quantities like speed and acceleration magnitude are decodable from the very earliest layers. However, motion direction requires global pooling and only becomes accessible at the PEZ. This progression closely mirrors the visual processing hierarchy found in the primate brain (moving from localized V1 neurons to position-invariant MT neurons).

Key Results

  • Sharp Performance Jumps: Linear probe accuracy on the IntPhys task transitions abruptly from chance (~50%) to peak performance (~85–95%) right at the PEZ boundary.
Figure 1. The Physics Emergence Zone consistently emerges one-third in the model’s layers. We probe performance on V-JEPA 2 (Large, Huge, and Giant) and VideoMAEv2-G across all layers for the possible-vs-impossible physical reasoning task. The shaded area is the emergence zone one-third through the network, where the network starts performing well on the task for linear probes (left) and attentive-MLP probes (right). For full results, including for the VideoMAE-v2 family, see Appendix C.1.
  • Orthogonal Subspaces: The projection overlap between the direction subspace and the IntPhys subspace ranges between 7% and 13%, which perfectly matches the baseline expected for completely random, independent subspaces of the same dimensionality.
  • Causal Attention Substrate: Suppressing spatiotemporally local attention heads exclusively inside the PEZ severely degrades direction decoding ($R^2$ drops from 0.97 to 0.14) and intuitive physics accuracy, while leaving static image classification on ImageNet entirely unaffected.
  • High-Dimensional Steering: Unlike language models where semantic concepts can often be manipulated along a single vector, motion direction is encoded as a ring-shaped population code that resists low-dimensional steering. Successfully shifting the decoded direction requires a coordinated intervention across 20 to 50 orthogonal dimensions.

Limitations and Future Work

  • Architecture Constraints: The analysis is restricted to encoder-based video transformers trained with masked prediction objectives; autoregressive or diffusion-based world models might exhibit entirely different internal structures.
  • Simplified Physics: The diagnostic tasks focus on coarse spatiotemporal continuity and basic kinematics (velocity/acceleration). They do not isolate more intricate physical properties like friction, fluid dynamics, or complex multi-object causal chains.
  • Synthetic Gap: The fine-grained motion variables were analyzed using highly controlled synthetic datasets, which may not fully capture how physical properties are encoded in messy, natural videos.

Summary updated

Abstract

A long-standing question in physical reasoning is whether video-based models need to rely on factorized representations of physical variables in order to make physically accurate predictions, or whether they can implicitly represent such variables in a task-specific, distributed manner. While modern video world models achieve strong performance on intuitive physics benchmarks, it remains unclear which of these representational regimes they implement internally. Here, we present the first interpretability study to directly examine physical representations inside large-scale video encoders. Using layerwise probing, subspace geometry, patch-level decoding, and targeted attention ablations, we characterize where physical information becomes accessible and how it is organized within encoder-based video transformers. Across architectures, we identify a sharp intermediate-depth transition -- which we call the Physics Emergence Zone -- at which physical variables become accessible. Physics-related representations peak shortly after this transition and degrade toward the output layers. Decomposing motion into explicit variables, we find that scalar quantities such as speed and acceleration are available from early layers onwards, whereas motion direction becomes accessible only at the Physics Emergence Zone. Notably, we find that direction is encoded through a high-dimensional population structure with circular geometry, requiring coordinated multi-feature intervention to control. These findings suggest that modern video models do not use factorized representations of physical variables like a classical physics engine. Instead, they use a distributed representation that is nonetheless sufficient for making physical predictions.
BibTeX
@inproceedings{joseph2026interpreting,
  author = {Sonia Joseph and Quentin Garrido and Randall Balestriero and Matthew Kowal and Thomas Fel and Shahab Bakhtiari and Blake Richards and Mike Rabbat},
  title = {{Interpreting Physics in Video World Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=aijGVmEG9Y},
}
Figure