TL;DR
HERA (Historical Evidence Routing Adapter) is a lightweight, parameter-efficient adapter designed to help latent video world models remember occluded objects without disrupting their pretrained pathways. By separating memory storage, selective retrieval, and workspace integration, HERA significantly improves a frozen V-JEPA 2 model's ability to reason about physical continuity and object permanence.
What Problem Does This Paper Address?
Latent world models like V-JEPA 2 are excellent at predicting future video dynamics in a self-supervised manner. However, they suffer from an "out of sight, out of mind" problem when dealing with physical events under occlusion. When an object moves behind an occluder, its essential properties—like color, shape, or initial trajectory—get diluted over time as new visual tokens flood the model's context window.
Existing solutions typically try to fix this by expanding the temporal context window (which dilutes early sparse evidence) or forcing rigid object-centric "slots" (which are difficult to reliably extract from frozen representations). The core challenge the authors tackle is: How can we selectively preserve and route early historical evidence to a frozen predictor exactly when it becomes relevant, without interfering with its native latent workspace?
Methods
The authors introduce HERA, instantiated with a mechanism called Register-Routed Patch Memory (RRPM). Crucially, the base video model (V-JEPA 2-G) remains completely frozen. HERA adds only 3.00M trainable parameters divided across three specialized modules:
- Structured Memory Bank: Instead of caching every single patch token redundantly, it groups context tokens into four distinct temporal roles:
- Anchor: Captures the earliest frames (initial object identity, color, position).
- Middle: Compresses intermediate motion and interaction history.
- Recent: Preserves the state immediately prior to the prediction window.
- Global: Provides a coarse summary of the entire context.
- Memory Registers: Trainable tokens dedicated to querying the Structured Memory Bank via gated cross-attention to pull out relevant historical details.
- Workspace Registers: A separate set of tokens that take the retrieved information and integrate it directly into the frozen predictor's native self-attention pathway.
By separating the retrieval mechanism (Memory Registers) from the internal computation mechanism (Workspace Registers), HERA avoids rewriting or corrupting the frozen predictor's native representations.
Datasets
- Physion: A dataset containing 14,000 videos focused on physical prediction, used exclusively for training the HERA adapter modules.
- IntPhys2 (Main split): A benchmark containing 1,012 videos organized into 506 matched pairs used for evaluation. It leverages a "violation-of-expectation" paradigm, pairing a physically possible video with an impossible one (e.g., an object magically changing color or disappearing behind an occluder).
Key Insights and Ideas
- Routing > Storing: Simply expanding a model's memory buffer is insufficient. The bottleneck in physical reasoning isn't just retaining history, but selectively routing the right piece of history to the predictor at the exact moment an occlusion or collision occurs.
- Temporal Asymmetry Protects Evidence: Grouping memory by temporal roles (like "Anchor" vs. "Recent") prevents crucial early object properties from being overwritten or diluted by recent, less informative background motion.
- An Elegant Adapter Strategy: You don't need to fine-tune massive world models or build complex object-tracking pipelines to achieve physical memory. A lightweight register-based routing mechanism attached to a frozen backbone is highly effective and computationally cheap.
Key Results
The evaluation uses a metric called pairwise AvgSurprise accuracy, which tests whether the model is more "surprised" (yields a higher L1 prediction error in latent space) by a physically impossible video than its possible counterpart.
- Overall Performance: HERA with RRPM improved the baseline V-JEPA 2-G pairwise accuracy from 52.57% to 54.35% on the IntPhys2 Main split.
- Subgroup Breakthroughs: The model achieved massive gains in specific physical reasoning categories where tracking hidden objects is critical:
- Fixed-camera continuity: Jumped from 46.15% to 57.69% (+11.54% gain).
- Fixed-camera immutability (remembering an object's properties): Rocketed from 46.15% to 63.46% (+17.31% gain).
- Solidity: Improved from 47.26% to 55.48% (+8.22% gain).
- Ablation Validation: Removing either the Memory Registers or the Workspace Registers dropped performance back down to ~53.56%, confirming that separating retrieval from workspace integration is vital.
Limitations and Future Work
- Room for Growth: While HERA shows strong relative improvements, an overall accuracy of 54.35% on IntPhys2 indicates that physical world modeling remains an incredibly challenging task with significant room for baseline improvement.
- Layout Dependencies: The temporal partitioning mechanism assumes a relatively regular and valid layout of video tubelets. If the context mask is highly irregular, the model has to fall back to a global-only summary, losing its precise temporal structuring.
- Future Directions: The authors suggest extending HERA to handle richer, more explicitly structured states and testing it across broader, more complex physical environments beyond benchmark settings.
Summary updated