TL;DR
Standard video world models fail when an action is undone because they assume the newest memory is always the truest. ReDate fixes this by using agent action logs to assign "valid times" to memories, allowing "frozen" (unretrained) models to correctly render reverted states by simply updating metadata.
What Problem Does This Paper Address?
Interactive video world models act as "simulators" that predict what an agent will see next based on its actions and a memory bank of past observations. Traditionally, these models use capture time (when a frame was recorded) to order their memories. They follow a "latest-timestamp-wins" rule: if the model has two views of the same area, it trusts the newer one.
This logic breaks in dynamic environments where actions are frequently undone. For example, if an agent opens a drawer, records a frame, and then closes it, the "closed" state is now the truth. But if the agent returns to that drawer later without seeing it closed, the newest memory in its bank shows it open. Standard models will incorrectly render the drawer as open. Existing solutions either delete old frames (losing valuable visual data) or retrain massive models. The authors argue that "old is not false"—an old observation can become valid again if the world state reverts.
Methods
The authors introduce ReDate, a training-free approach that borrows a concept from temporal databases: separating transaction time (capture time) from valid time (when the content actually holds).
- Action Log Tracking: ReDate maintains a state tracker that reads the agent's action log (e.g., "picked up pillow," "closed door").
- Valid-Time Re-dating: When an action restores a region to a previous state (an "undo"), ReDate identifies the old frame that matches this state and updates its timestamp to the current time.
- Per-Token Granularity: Because a single frame might contain both current and outdated regions, ReDate operates at the token level. It moves only the "valid" tokens to the present while pushing "outdated" tokens (like a pillow that is no longer there) into the distant past.
- Leveraging Frozen Generators: Since world model generators (like FAR or HY-WorldPlay) are already trained to prefer newer timestamps, ReDate works out-of-the-box without any weight updates. It simply changes the "metadata" the generator sees.
Datasets
- AI2-THOR: The primary benchmark for evaluation, specifically using the FAR benchmark which includes 1,309 test clips of agents interacting with household objects.
- EPIC-KITCHENS-100: Used to provide real-world evidence for the "undo" problem. Analysis showed that over 50% of common kitchen actions (opening cupboards, turning on switches) are eventually undone.
- Charades: Real-world home videos used to test how temporal positioning steers models like HY-WorldPlay after an undo in non-simulated environments.
Key Insights and Ideas
- The "Latest-Timestamp" Bias: The authors prove that frozen generators inherently act as "soft latest-timestamp" rules. ReDate exploits this existing behavior rather than trying to fight it or retrain it.
- Information Asymmetry: Without a fresh visual observation, the only way a model can know a state has reverted is through the action log. ReDate provides the missing link between symbolic action logs and visual memory.
- Keeping the Frame Whole: Unlike deletion-based methods that remove "false" frames and leave blurry "ghosts" or holes in the background, ReDate's per-token movement keeps the background stable while only updating the dynamic parts of the scene.
Key Results
- Accuracy Boost: On "reverted" steps (after an undo), ReDate raises the accuracy of the FAR model from 0.36 to 0.62.
- Generalizability: It significantly improves HY-WorldPlay, raising its share of correct states from a near-zero 0.003 to 0.19.
- State Metric: The authors proposed a new calibrated state metric using LPIPS in latent space, which more accurately distinguishes between current, outdated, and ambiguous renders than standard pixel-based metrics.
- Efficiency: ReDate adds negligible overhead (approx. 0.03s per rollout step) and requires no additional GPU memory for training.
Limitations and Future Work
- Log Dependency: ReDate relies on an accurate action log. If the agent's internal state tracker fails or the log is incomplete, the re-dating will be incorrect.
- Agent-Centric: The current model assumes only the agent changes the scene. Handling "dynamic" worlds where other actors (NPCs or humans) change things out of view remains an open challenge.
- Unseen Reversions: If the agent reverts a state but never captured a frame of that state from its current viewpoint, ReDate still struggles. The authors suggest "rendering a valid-time view" from other poses as a future direction.
Summary updated