TL;DR
Video world models often fail "memory" tests not because they forget, but because they never executed the required action in the first place. This paper introduces MIST, a diagnostic evaluation that separates scene setup from state reveal, and M3-Agent, a test-time revision loop that triples performance by explicitly "making the missing move" in the model's instructions.
What Problem Does This Paper Address?
When we test a video model's "memory"—for example, asking it to show a child sitting in a chair after the camera has looked away—we usually look at the final frame. If the chair is empty, we assume the model "forgot" the child was there.
However, the authors argue that this is a diagnostic error. Often, the model fails because it never actually showed the child sitting down before the camera moved. If the transition never happened, there is no "memory" to test. Current benchmarks conflate execution failures (failing to perform an action) with memory failures (failing to persist a state). This paper aims to decouple these two issues to find out where video world models are actually breaking down.
Methods
The authors propose two major contributions:
- MIST (Memory Evaluation with Delayed Reveal): Instead of one continuous clip, MIST uses a three-shot structure to isolate variables:
- Shot 1 (Setup): Establishes the initial scene.
-
Shot 2 (Gap): An opaque transition where the change is supposed to happen "off-camera."
- Shot 3 (Reveal): The model is asked to show the result of the change. By scoring these "gates" (Setup, Leak, Reveal, and Inside) separately, researchers can pinpoint exactly when the model fails.
-
M3-Agent (Make the Missing Move): This is a test-time agent designed to fix execution errors without retraining the underlying "frozen" video generator. It follows a recursive loop:
- Diagnose: A Vision-Language Model (VLM) inspects the failed output and identifies the failure mode (e.g., "no action," "wrong binding").
- Revise: The agent rewrites the input instruction to be more explicit about the specific move and the visual evidence required in the final frame.
- Generate & Select: The model generates new candidates, and the agent keeps the one that best satisfies the instruction.
Datasets
The study evaluates several state-of-the-art video generators, including Wan TI2V-5B, Wan T2V-A14B, and MiniMax-H3. They test these models across three primary settings:
- MIST (Internal): A new diagnostic suite of 43 cases focusing on presence, count, binding, and history updates.
- WRBench: An official benchmark for physical understanding.
- MBench-T: A benchmark specifically for memory capability in video world models.
Key Insights and Ideas
- Execution is the Bottleneck: In the baseline Wan TI2V-5B model, 41 out of 43 MIST cases failed to show the requested change. The problem wasn't that the model "forgot" the state; it simply never opened the box or sat the person down.
- Naming the Target Isn't Enough: Simply telling the model "there is an orange marble in the tin" helps less than explicitly revising the action (e.g., "open the tin to reveal the orange marble").
- The "Aha" Moment: Many "memory" failures on official benchmarks like WRBench are actually just the camera failing to complete its path (e.g., it never "returns" to the object).
Key Results
The M3-Agent significantly improves performance across the board by forcing the model to execute correctly:
- MIST Reveal Score: More than tripled, rising from 0.147 to 0.492.
- WRBench (Score T): Increased by 26.1% (0.452 → 0.570). The authors found that 56% of this rise came from simply fixing the "departure" check (ensuring the camera actually left and returned).
- MBench-T (Score U): Increased by 52.7% (0.292 → 0.446).
Limitations and Future Work
- Oracle Selection: The "Keep-best" strategy currently relies on an "oracle" (the same scorer used for evaluation) to pick the best candidate. A truly deployable agent would need a "target-blind" critic that can judge quality without knowing the "gold" answer.
- Memory vs. Hallucination: While M3-Agent makes the model show a result, it doesn't prove the model actually stored the state in its hidden layers. It might just be very good at hallucinating a plausible result based on the revised prompt.
- Future Direction: The authors suggest that future memory evaluations must publish setup, execution, and reveal scores separately to avoid misleading "forgetting" labels.
Summary updated