Counterfactual World Models: JEPAs That Reason About "What If"

Randall Balestriero, Yann LeCun, Kerda VarakuarXiv 2026 · arXiv

TL;DR

Researchers have developed a way for world models to answer "what if" questions—like "where would my boat have gone if I'd steered left instead of right in this specific hidden current?"—without needing any extra data or retraining. By leveraging the physical principle that "nature makes no leaps," they show that simply reusing the prediction error (residual) from an observed event allows standard Joint-Embedding Predictive Architectures (JEPAs) to predict individual counterfactuals with remarkable accuracy.

What Problem Does This Paper Address?

Imagine a boat trying to cross a lake. The pilot chooses a heading, but an invisible underwater current pushes the boat off course. A standard world model can predict where the boat might land on average, but it can't know the specific current of the moment until it sees the result.

Once the boat lands, we have a "factual" observation. But what if we want to debug a failure? We need to know: "Would a different heading have worked in that exact same current?" This is an individual counterfactual.

Standard world models are great at predicting average outcomes (interventions), but they struggle with individual counterfactuals. The authors prove a "nonidentification" theorem: factual data alone—no matter how much you have—cannot distinguish between different possible "what if" worlds. To solve this, you usually need a perfect simulator or paired data (seeing two different actions in the same hidden conditions), both of which are rare in the real world.

Methods

The authors introduce a surprisingly simple technique called residual reuse. It is based on the principle natura non facit saltus (nature makes no leaps). In many physical systems, actions (like steering) and disturbances (like currents) act as separate forces that only interact over time.

For short time steps, the interaction between these forces is negligible. This leads to a simple mathematical "hack": to predict a counterfactual outcome, take the observed outcome and add the difference between the model's prediction for the new action and its prediction for the original action.

The residual reuse principle: the observed outcome carries information about the hidden event, while the world model carries the effect of the action.

This method requires:

  1. A frozen encoder: Like DINOv2 or MAE, to turn images into latent features.
  2. A standard JEPA: A world model trained only on factual transitions to predict the next latent state.
  3. No retraining: The counterfactual reasoning happens entirely at inference time.

Datasets

The researchers validated their approach across seven different simulated physical environments:

  • HandReach: A robotic hand moving its wrist and fingers.
  • Ball-in-Cup: A classic toy task.
  • FetchReach: A robotic arm reaching for a target.
  • PassWater: A task involving liquid transport.
  • BlueROV2: An underwater vehicle subject to complex hydrodynamics.
  • Push-T & Racetrack: Used as diagnostic environments to test the limits of the theory.

They used a variety of high-performance frozen vision encoders, including DINOv2, DINOv3, MAE, SigLIP 2, CLIP, and EVA-02.

Key Insights and Ideas

  • The "Residual" is the Key: The difference between what a model predicted and what actually happened is a "fingerprint" of the hidden disturbance. Because physics is often additive over short windows, this fingerprint can be "reused" to see how a different action would have played out.
  • Physics as a Causal Prior: Instead of trying to learn causal relationships from scratch (which is mathematically impossible from factual data alone), we can use the structure of Newtonian mechanics as a "prior" that identifies the counterfactual.
  • Hindsight Planning: This technique allows for "hindsight planning"—looking back at a failed attempt and finding the specific action that would have reached the goal in that exact situation. This is much more effective than just trying a different "average" good action.

Key Results

The results show a massive leap in counterfactual accuracy compared to standard models:

  • Accuracy Boost: On four primary environments, the counterfactual $R^2$ (a measure of how much variance the model explains) jumped from 38–46% for standard models to 86–96% using residual reuse.
  • Hindsight Success: In planning tasks, the counterfactual model found the successful action in 35–69% of cases. For comparison, an "oracle" planner that knew the physics but didn't see the specific outcome only succeeded 14–23% of the time.
  • Robustness: The method works across seven different state-of-the-art vision encoders, proving that the principle is robust to how the world is represented.
Comparison of counterfactual $R^2$ across different environments. The colored bars (ours) consistently outperform the black ticks (standard predictors).

Limitations and Future Work

While powerful, the "nature makes no leaps" assumption has its limits:

  • Multiplicative Faults: If a disturbance changes the effectiveness of your action (e.g., a motor that only gives 50% power), the additive assumption fails.
  • Discontinuities: Sudden events like collisions (contacts) break the smooth, short-time approximation.
  • Coordinate Folds: The method requires "good" coordinates. If your representation "folds" the world (like using $cos(\theta)$ where two different angles look the same), the math breaks down.

Future Work: The authors suggest training encoders or readouts specifically to satisfy these additive properties, potentially expanding the "operating window" where counterfactual reasoning is safe and accurate.

Summary updated

Abstract

World models operate in a world they cannot fully observe. When a hidden disturbance, such as a current pushing a boat, changes the outcome of an action, a counterfactual question follows: what would have happened under a different action, subject to the same disturbance? Its answer separates bad decisions from bad luck. We prove that world models trained on factual transitions cannot answer it on their own, whatever their data or capacity. For many physical systems, the observed outcome and a simple property of the dynamics suffice: natura non facit saltus, nature makes no leaps. When actions and disturbances act as separate forces, their interaction is second-order in time, so the counterfactual outcome is, to first order, the observed outcome plus the difference between the world model's predictions under the alternative and factual actions. This yields a counterfactual world model with no counterfactual data, extra parameters, or retraining. For transitions of duration $h$, we prove an $O(h^2)$ bound on the counterfactual error relative to the disturbance variance, establish a matching minimax lower bound from unpaired factual data, and characterize failures at discontinuities, coordinate folds, and disturbances that scale the command. We validate these predictions from pixels with JEPA world models on frozen DINOv2 and masked autoencoder (MAE) encoders. On four rendered simulators, our approach improves counterfactual $R^2$ from $38$--$46\%$ to $86$--$96\%$ with the same world models; on a fifth, an underwater vehicle, it reaches $90$--$94\%$ with sixteen times more factual data. Four further frozen encoders, DINOv3, SigLIP~2, CLIP and EVA-02, give $81$--$95\%$ on the same simulators. In hindsight planning, it finds the action that would have reached the goal in $35$--$69\%$ of events, against $14$--$23\%$ for an oracle planner that knows the simulator but not the outcome.
BibTeX
@misc{balestriero2026counterfactual,
  author = {Randall Balestriero and Yann LeCun and Kerda Varaku},
  title = {{Counterfactual World Models: JEPAs That Reason About "What If"}},
  year = {2026},
  eprint = {2611.24778},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24778},
}
Figure