LATER: Learning Action-Conditioned Temporal Evidence Retrieval for Predictive World Models

Zihan Zhou, Wenxi Xu, Qilin Wang, Hao TangarXiv 2026 · arXiv

TL;DR

LATER (Learning Action-Conditioned Temporal Evidence Retrieval) is a new world model architecture that mimics human working memory by separating the storage of physical evidence from its retrieval. By using a partitioned memory bank and an action-conditioned "response bridge," LATER allows robots to remember hidden physical properties (like friction) discovered during early interactions and use that knowledge to plan actions much later, even after the evidence is no longer visible.

What Problem Does This Paper Address?

Most modern world models, such as Joint-Embedding Predictive Architectures (JEPAs), rely on a finite window of recent visual history to predict the future. This creates a "memory bottleneck": if a robot performs an exploratory action—like pushing an object to sense its weight or friction—and then waits or moves away, that crucial physical evidence can be "washed out" of its recent history.

When the robot needs to interact with that object again, it has effectively forgotten what it learned. The challenge is twofold: how to retain interaction evidence through intervening observations and how to read that evidence selectively to evaluate candidate actions during planning without letting hypothetical "imagined" outcomes overwrite real-world observations.

Methods

LATER builds upon the CF-JEPA backbone, which factorizes latent space into controllable (action-influenced) and action-adversarial (background/noise) streams. The authors introduce several key innovations inspired by cognitive science:

  • Partitioned Memory Bank: LATER uses two recurrent "writers" (GRUs) to store completed real-world transitions. Crucially, these memories remain frozen during the planning phase (candidate rollouts). This ensures that the robot's "imagination" doesn't corrupt its actual memories of the world.
  • Action-Conditioned Response Bridge: Instead of a generic history, LATER uses a "response bridge" that queries the memory using both the current state and a candidate action. This allows the model to ask, "Based on what I saw earlier, how will this specific action turn out?"
  • Zero-Gated Residuals: To ensure stable training, the connection between the memory and the predictor starts at zero. This allows the model to maintain baseline performance while slowly learning to incorporate memory.
  • Multi-cut Auxiliary Supervision: Since the gates are initially closed, the authors use an auxiliary training head to provide a direct gradient path to the memory writers, ensuring they learn to encode useful information from the start.
Figure 2. Architecture of LATER. (a) The trainable CF backbone receives recent tokens and zero-gated memory inputs. The bridge corrects only the controllable prediction; no pixel decoder is used. (b) Separate GRU writers summarize completed real transitions. Both banks stay fixed during candidate rollouts. (c) A state–action query reads both banks for a gated rank-8 residual (Eqs. (7)–(8)). Multi-cut supervision trains the writers; its head is discarded for planning.

Datasets

The authors evaluated LATER on several standard and newly synthesized benchmarks:

  • Standard Tasks: Reacher, PushT, TwoRoom, and single-Cube.
  • Synthesized Multi-Stage Tasks: The authors created Two-Stage PushT, Directional TwoRoom, and Dual Cube. These tasks are specifically designed to require "delayed evidence retrieval"—an initial "probe" phase reveals a property (like friction or a passage rule), followed by a reset or wait period, and finally a "query" phase where that information is needed for success.

Key Insights and Ideas

  • Retention vs. Readout: The core "aha moment" is the functional dissociation between maintaining information and selecting it for behavior. By decoupling these, LATER avoids the interference that plagues standard recurrent models.
  • Action-Gated Retrieval: Physical properties like friction are often "latent" until an action is applied. LATER’s bridge (Figure 2c) recognizes that the same memory (e.g., "this block is heavy") is only relevant when the robot is actually trying to push it, not when it's just waiting.
  • Frozen Memories for Counterfactuals: During planning, a robot might simulate hundreds of possible futures. If the memory updated during these simulations, it would quickly become a mess of "what ifs." Keeping the memory bank fixed to real-world observations is vital for accurate long-term planning.
Figure 3. Matched simulator examples. PushT E5 (top) and TwoRoom E10 (bottom) share initial state, goal and distractors within each row. These are the first complete cases solved by LATER but not the displayed controls. LeWM/SMWM are matched task-trained adaptations.

Key Results

LATER significantly outperforms state-of-the-art baselines, especially in environments with visual distractors:

  • Success Rate (SR): On the "distracted" Two-Stage PushT task, LATER improved terminal success from 15.5% to 25.0% over the standard CF-JEPA. In Directional TwoRoom, the first-hit SR rose from 15.0% to 26.5%.
  • Prediction Accuracy: LATER achieved the lowest mean physical readout error across six different models. It reduced the mean absolute error (MAE) during pushing by 20.2% relative to CF-JEPA.
  • Robustness: In a "stress test" using the Reacher task, a model trained with only 2 distractors maintained a 94–100% success rate even when the number of distractors was scaled up to 64.

Limitations and Future Work

While LATER shows a marked improvement, the absolute success rates on the most difficult tasks (like Two-Stage PushT) remain relatively low (25%), suggesting that multi-stage physical reasoning is still a very hard problem.

The authors note that while LATER handles "delayed" evidence well, future work could extend this to even longer interactions or "lifelong" learning scenarios. Additionally, the current memory is updated at every real transition; exploring "event-triggered" consolidation (only remembering surprising or important events) could further improve efficiency and long-term retention.

Summary updated

Abstract

Early interactions can reveal physical properties that later observations no longer show. These properties can still gov- ern the effects of future actions. Inspired by the distinction between retention and selective readout in working mem- ory and cognitive science, we introduce LATER, an action- conditioned world model built on CF-JEPA. LATER stores completed real transitions in partitioned memories that remain fixed during candidate rollouts. A response bridge queries these memories using the current state and candidate action, while multi-cut auxiliary supervision trains the memory writ- ers. On distracted Two-Stage PushT, LATER raises terminal success from 15.5% to 25.0% over reconstructed CF-JEPA. In a separate comparison across six models and two scratch- training seeds, LATER achieves the lowest mean physical readout error in both clean and distracted views, reducing along-push mean absolute error (MAE) under distractors by 20.2% relative to CF-JEPA. Controlled history interventions show that later predictions depend on earlier physical evi- dence even when query observations and actions are fixed, with larger effects during pushing than waiting.
BibTeX
@misc{zhou2026later,
  author = {Zihan Zhou and Wenxi Xu and Qilin Wang and Hao Tang},
  title = {{LATER: Learning Action-Conditioned Temporal Evidence Retrieval for Predictive World Models}},
  year = {2026},
  eprint = {2611.25671},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25671},
}
Figure