Execution Before Memory: Diagnosing Memory Failures in Video World Models

Weiyang Jin, Jiwen Yu, Sihan Xu, Yuwei Niu, Xihui LiuarXiv 2026 · arXiv

TL;DR

Video world models often fail "memory" tests not because they forget, but because they never executed the required action in the first place. This paper introduces MIST, a diagnostic evaluation that separates scene setup from state reveal, and M3-Agent, a test-time revision loop that triples performance by explicitly "making the missing move" in the model's instructions.

What Problem Does This Paper Address?

When we test a video model's "memory"—for example, asking it to show a child sitting in a chair after the camera has looked away—we usually look at the final frame. If the chair is empty, we assume the model "forgot" the child was there.

However, the authors argue that this is a diagnostic error. Often, the model fails because it never actually showed the child sitting down before the camera moved. If the transition never happened, there is no "memory" to test. Current benchmarks conflate execution failures (failing to perform an action) with memory failures (failing to persist a state). This paper aims to decouple these two issues to find out where video world models are actually breaking down.

Methods

The authors propose two major contributions:

  1. MIST (Memory Evaluation with Delayed Reveal): Instead of one continuous clip, MIST uses a three-shot structure to isolate variables:
  2. Shot 1 (Setup): Establishes the initial scene.
  3. Shot 2 (Gap): An opaque transition where the change is supposed to happen "off-camera."

    • Shot 3 (Reveal): The model is asked to show the result of the change. By scoring these "gates" (Setup, Leak, Reveal, and Inside) separately, researchers can pinpoint exactly when the model fails.
  4. M3-Agent (Make the Missing Move): This is a test-time agent designed to fix execution errors without retraining the underlying "frozen" video generator. It follows a recursive loop:

  5. Diagnose: A Vision-Language Model (VLM) inspects the failed output and identifies the failure mode (e.g., "no action," "wrong binding").
  6. Revise: The agent rewrites the input instruction to be more explicit about the specific move and the visual evidence required in the final frame.
  7. Generate & Select: The model generates new candidates, and the agent keeps the one that best satisfies the instruction.
The M3-Agent workflow: diagnosing failures and revising instructions to ensure the "missing move" is executed.

Datasets

The study evaluates several state-of-the-art video generators, including Wan TI2V-5B, Wan T2V-A14B, and MiniMax-H3. They test these models across three primary settings:

  • MIST (Internal): A new diagnostic suite of 43 cases focusing on presence, count, binding, and history updates.
  • WRBench: An official benchmark for physical understanding.
  • MBench-T: A benchmark specifically for memory capability in video world models.

Key Insights and Ideas

  • Execution is the Bottleneck: In the baseline Wan TI2V-5B model, 41 out of 43 MIST cases failed to show the requested change. The problem wasn't that the model "forgot" the state; it simply never opened the box or sat the person down.
  • Naming the Target Isn't Enough: Simply telling the model "there is an orange marble in the tin" helps less than explicitly revising the action (e.g., "open the tin to reveal the orange marble").
  • The "Aha" Moment: Many "memory" failures on official benchmarks like WRBench are actually just the camera failing to complete its path (e.g., it never "returns" to the object).
Visual examples of execution failures: the child never sits, the candy is never broken, and the box stays closed until the instruction is revised.

Key Results

The M3-Agent significantly improves performance across the board by forcing the model to execute correctly:

  • MIST Reveal Score: More than tripled, rising from 0.147 to 0.492.
  • WRBench (Score T): Increased by 26.1% (0.452 → 0.570). The authors found that 56% of this rise came from simply fixing the "departure" check (ensuring the camera actually left and returned).
  • MBench-T (Score U): Increased by 52.7% (0.292 → 0.446).
Analysis of WRBench improvements, showing that the majority of the score increase comes from fixing the camera path (D5 and D6 gates).

Limitations and Future Work

  • Oracle Selection: The "Keep-best" strategy currently relies on an "oracle" (the same scorer used for evaluation) to pick the best candidate. A truly deployable agent would need a "target-blind" critic that can judge quality without knowing the "gold" answer.
  • Memory vs. Hallucination: While M3-Agent makes the model show a result, it doesn't prove the model actually stored the state in its hidden layers. It might just be very good at hallucinating a plausible result based on the revised prompt.
  • Future Direction: The authors suggest that future memory evaluations must publish setup, execution, and reveal scores separately to avoid misleading "forgetting" labels.

Summary updated

Abstract

A memory test for a video world model should say whether the scene was set, whether the asked change was carried out, and whether the last frame shows the result. Current tests fold these into one number, so a low score is easy to read as forgetting. When the change is not carried out, the hidden state transition cannot appear, so the test is not really a test of memory. We introduce MIST, an evaluation that separates the change from the memory it should leave behind. The model first sets the scene while the change stays hidden, and only later is asked to show it, with leakage before that point kept under control. This separation locates the bottleneck: the model sets the scene but does not carry out the instructed change. The same gap appears in general memory evaluations. On that basis we propose M3-Agent (Make the Missing Move), which recursively revises the details of the input instruction so that the model successfully carries out the requested instruction. Relative to the baseline, it more than triples the MIST reveal score. On the benchmarks, WRBench's score T rises by 26.1% and MBench-T's score U by 52.7%. The method meets the problem from the execution side: it shows what the frozen model can do once the move is stated, and it sets a stricter baseline.
BibTeX
@misc{jin2026execution,
  author = {Weiyang Jin and Jiwen Yu and Sihan Xu and Yuwei Niu and Xihui Liu},
  title = {{Execution Before Memory: Diagnosing Memory Failures in Video World Models}},
  year = {2026},
  eprint = {2610.23542},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23542},
}
Figure