HERA: Historical Evidence Routing Adapter for Physical Prediction in Latent World Models

Yuanruyi, Yue Cao, Haojia Gao, Guanqiu Guo, Ziyuezhang, Shangqin, Junbo Tan, Bokui Chen, Zhuo Zou, Xueqian WangarXiv 2026 · arXiv

TL;DR

HERA (Historical Evidence Routing Adapter) is a lightweight, parameter-efficient adapter designed to help latent video world models remember occluded objects without disrupting their pretrained pathways. By separating memory storage, selective retrieval, and workspace integration, HERA significantly improves a frozen V-JEPA 2 model's ability to reason about physical continuity and object permanence.

What Problem Does This Paper Address?

Latent world models like V-JEPA 2 are excellent at predicting future video dynamics in a self-supervised manner. However, they suffer from an "out of sight, out of mind" problem when dealing with physical events under occlusion. When an object moves behind an occluder, its essential properties—like color, shape, or initial trajectory—get diluted over time as new visual tokens flood the model's context window.

Existing solutions typically try to fix this by expanding the temporal context window (which dilutes early sparse evidence) or forcing rigid object-centric "slots" (which are difficult to reliably extract from frozen representations). The core challenge the authors tackle is: How can we selectively preserve and route early historical evidence to a frozen predictor exactly when it becomes relevant, without interfering with its native latent workspace?

Methods

The authors introduce HERA, instantiated with a mechanism called Register-Routed Patch Memory (RRPM). Crucially, the base video model (V-JEPA 2-G) remains completely frozen. HERA adds only 3.00M trainable parameters divided across three specialized modules:

  1. Structured Memory Bank: Instead of caching every single patch token redundantly, it groups context tokens into four distinct temporal roles:
  2. Anchor: Captures the earliest frames (initial object identity, color, position).
  3. Middle: Compresses intermediate motion and interaction history.
  4. Recent: Preserves the state immediately prior to the prediction window.
  5. Global: Provides a coarse summary of the entire context.
  6. Memory Registers: Trainable tokens dedicated to querying the Structured Memory Bank via gated cross-attention to pull out relevant historical details.
  7. Workspace Registers: A separate set of tokens that take the retrieved information and integrate it directly into the frozen predictor's native self-attention pathway.

By separating the retrieval mechanism (Memory Registers) from the internal computation mechanism (Workspace Registers), HERA avoids rewriting or corrupting the frozen predictor's native representations.

Figure 1: Architecture of HERA with RRPM. A frozen V-JEPA 2 context encoder produces tokens that are organized into temporally structured memory. Memory Registers retrieve this evidence through gated cross-attention, while Workspace Registers support its integration through the frozen predictor’s self-attention pathway.
Figure 2: Structured Memory Bank in RRPM. The Structured Memory Bank preserves historical patch evidence according to temporal role. Anchor memory stores early object evidence, middle memory summarizes motion and interaction history, recent memory stores the state immediately before prediction, and global memory provides a coarse context summary. The bank gives the predictor access to physically relevant history without requiring object supervision.

Datasets

  • Physion: A dataset containing 14,000 videos focused on physical prediction, used exclusively for training the HERA adapter modules.
  • IntPhys2 (Main split): A benchmark containing 1,012 videos organized into 506 matched pairs used for evaluation. It leverages a "violation-of-expectation" paradigm, pairing a physically possible video with an impossible one (e.g., an object magically changing color or disappearing behind an occluder).

Key Insights and Ideas

  • Routing > Storing: Simply expanding a model's memory buffer is insufficient. The bottleneck in physical reasoning isn't just retaining history, but selectively routing the right piece of history to the predictor at the exact moment an occlusion or collision occurs.
  • Temporal Asymmetry Protects Evidence: Grouping memory by temporal roles (like "Anchor" vs. "Recent") prevents crucial early object properties from being overwritten or diluted by recent, less informative background motion.
  • An Elegant Adapter Strategy: You don't need to fine-tune massive world models or build complex object-tracking pipelines to achieve physical memory. A lightweight register-based routing mechanism attached to a frozen backbone is highly effective and computationally cheap.

Key Results

The evaluation uses a metric called pairwise AvgSurprise accuracy, which tests whether the model is more "surprised" (yields a higher L1 prediction error in latent space) by a physically impossible video than its possible counterpart.

  • Overall Performance: HERA with RRPM improved the baseline V-JEPA 2-G pairwise accuracy from 52.57% to 54.35% on the IntPhys2 Main split.
  • Subgroup Breakthroughs: The model achieved massive gains in specific physical reasoning categories where tracking hidden objects is critical:
  • Fixed-camera continuity: Jumped from 46.15% to 57.69% (+11.54% gain).
  • Fixed-camera immutability (remembering an object's properties): Rocketed from 46.15% to 63.46% (+17.31% gain).
  • Solidity: Improved from 47.26% to 55.48% (+8.22% gain).
  • Ablation Validation: Removing either the Memory Registers or the Workspace Registers dropped performance back down to ~53.56%, confirming that separating retrieval from workspace integration is vital.

Limitations and Future Work

  • Room for Growth: While HERA shows strong relative improvements, an overall accuracy of 54.35% on IntPhys2 indicates that physical world modeling remains an incredibly challenging task with significant room for baseline improvement.
  • Layout Dependencies: The temporal partitioning mechanism assumes a relatively regular and valid layout of video tubelets. If the context mask is highly irregular, the model has to fall back to a global-only summary, losing its precise temporal structuring.
  • Future Directions: The authors suggest extending HERA to handle richer, more explicitly structured states and testing it across broader, more complex physical environments beyond benchmark settings.

Summary updated

Abstract

Predictive video models have emerged as promising world models by learning latent visual dynamics from large-scale video. Yet these models remain challenged by physical events under occlusion, where later predictions may depend on object evidence that is no longer available in the current view. Addressing this challenge requires historical evidence not only to be preserved but also to remain accessible when it becomes relevant to a subsequent prediction. Existing approaches mainly enlarge the temporal context, cache generic video features, or impose explicit object-centric states, thereby improving the capacity or structure of retained history. However, they do not directly address how relevant historical evidence can be selectively retrieved and integrated into a pretrained predictor without interfering with its native latent workspace. Accordingly, we introduce HERA (Historical Evidence Routing Adapter), a framework for routing retained historical evidence into a frozen latent predictor, and instantiate it with Register-Routed Patch Memory (RRPM), a lightweight adapter comprising a Structured Memory Bank, Memory Registers, and Workspace Registers. On the IntPhys2 Main split, HERA with RRPM improves the pairwise AvgSurprise accuracy of V-JEPA 2-G from 52.57% to 54.35%. Subgroup analysis shows particularly strong improvements on fixed-camera continuity, from 46.15% to 57.69%, and fixed-camera immutability, from 46.15% to 63.46%. These results support historical evidence routing as a practical adaptation strategy for physical prediction in latent world models.
BibTeX
@misc{yuanruyi2026hera,
  author = {Yuanruyi and Yue Cao and Haojia Gao and Guanqiu Guo and Ziyuezhang and Shangqin and Junbo Tan and Bokui Chen and Zhuo Zou and Xueqian Wang},
  title = {{HERA: Historical Evidence Routing Adapter for Physical Prediction in Latent World Models}},
  year = {2026},
  eprint = {2608.05523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.05523},
}
Figure