TL;DR
HeMeR (Heterogeneous Memory Reconciliation) is an inference framework that allows embodied agents to selectively reuse their Key-Value (KV) cache even as their memory context dynamically evolves. By treating memory as structured chunks rather than a flat sequence, HeMeR significantly reduces redundant computation, cutting real-world robot inference time by over 35% while simultaneously improving task success rates.
What Problem Does This Paper Address?
Embodied agents—robots powered by Large Language Models (LLMs) or Vision-Language Models (VLMs)—don't just react to the present; they rely on a complex mix of memories. This includes declarative memory (facts about the world), procedural memory (learned skills/routines), and working memory (current observations and progress).
As a robot moves through a house to make coffee, its memory context is constantly shifting: new objects are seen, subgoals are completed, and different skill "adapters" (like LoRA) are swapped in and out. In a standard Transformer setup, even a tiny change to this context forces the model to recompute the entire Key-Value (KV) cache from scratch. This is computationally wasteful and creates a bottleneck for real-time interaction. Existing systems focus on what to store in memory, but HeMeR addresses the neglected problem of how to efficiently serve that memory during inference.
Methods
The authors introduce a "memory-serving" framework that anchors memory contents as structured chunks. Instead of a flat string of text, the model sees a sequence of chunks, each with metadata like a persistent ID, semantic role, and associated KV range.
HeMeR employs two primary mechanisms to reconcile these chunks with the existing cache:
- Chunk-Level KV Reconciliation: When the agent's state changes, HeMeR compares the new memory context to the old one. If a memory chunk is simply moved to a different position in the prompt, HeMeR uses RoPE (Rotary Position Embedding) remapping to update the cached keys' positional phase without recomputing the values. If a chunk is new or its content has changed, only that specific range is recomputed.
- Adapter-Stable KV Reconciliation: Embodied agents often use different LoRA adapters for different tasks (e.g., one for "picking" and one for "cleaning"). Switching adapters usually invalidates the cache. HeMeR tracks which adapter was used to compute which KV state and selectively recomputes only the ranges affected by an adapter switch, preserving compatible states across the transition.
Datasets
The researchers evaluated HeMeR across three distinct environments:
- ALFWorld: A text-based household simulator where agents must solve multi-step tasks using a discrete command interface.
- EmbodiedBench Habitat: A high-fidelity 3D simulator (using ReplicaCAD) where agents process RGB observations to perform tasks like "deposit apples on the TV stand."
- Real-world Robotics: A physical setup featuring a UR7e robotic arm performing long-horizon tasks like "Make a vanilla Americano and a slice of toast" in a tabletop kitchen-café environment.
Key Insights and Ideas
- Memory as a Serving Problem: The core "aha" moment is reframing heterogeneous memory use as a KV reconciliation problem. Efficiency isn't just about better retrieval; it's about how the model's internal state handles that retrieved data.
- Position Independence via RoPE: The realization that we can "teleport" cached states to new positions in a prompt using simple mathematical rotations (RoPE remapping) is a powerful tool for dynamic contexts.
- Cache Stability Equals Task Success: Interestingly, HeMeR doesn't just make robots faster; it makes them better. By preserving compatible KV states, the model maintains better internal consistency, leading to higher "active action ratios" (doing things that actually progress the task) and fewer redundant failures.
- Localization of Drift: While naive cache reuse can lead to "hidden-state drift" (where errors accumulate over time), HeMeR's schema-guided approach confines these deviations to a small number of affected chunks, keeping the overall model representation stable.
Key Results
HeMeR consistently outperformed strong baselines like MemRL and LRAgent across all benchmarks:
- ALFWorld: Achieved a 71.43% success rate (vs. 52.86% for MemRL) while slashing per-step latency from 0.90s to 0.18s.
- Habitat: Reached a 76.27% success rate, significantly higher than the best cache-reuse baseline (KEEP at 59.40%).
- Real-world: In physical robot trials, HeMeR achieved an 83.20% success rate and reduced per-step inference time from 2.94s to 1.87s.
Limitations and Future Work
The authors identify a few key areas for future exploration:
- Multi-Agent Systems: The current work focuses on a single agent. In multi-agent environments, concurrent updates to shared memory would require new concurrency control mechanisms to prevent cache inconsistency.
- Dependency on LoRA Design: The adapter-stable reconciliation relies on specific shared-attention LoRA designs. Extending this to more diverse adapter architectures remains an open challenge.
- Scaling: While HeMeR scales well to 128 memory chunks, extremely long-horizon tasks with thousands of memory fragments may require more aggressive eviction or compression strategies.
Summary updated