HeMeR: Heterogeneous Memory Reconciliation for Embodied Agents via Structured KV Reuse

Saehun Chun, Wonje Choi, Jinwoo Jang, Taeyoon Kwack, Honguk WooarXiv 2026 · arXiv

TL;DR

HeMeR (Heterogeneous Memory Reconciliation) is an inference framework that allows embodied agents to selectively reuse their Key-Value (KV) cache even as their memory context dynamically evolves. By treating memory as structured chunks rather than a flat sequence, HeMeR significantly reduces redundant computation, cutting real-world robot inference time by over 35% while simultaneously improving task success rates.

What Problem Does This Paper Address?

Embodied agents—robots powered by Large Language Models (LLMs) or Vision-Language Models (VLMs)—don't just react to the present; they rely on a complex mix of memories. This includes declarative memory (facts about the world), procedural memory (learned skills/routines), and working memory (current observations and progress).

As a robot moves through a house to make coffee, its memory context is constantly shifting: new objects are seen, subgoals are completed, and different skill "adapters" (like LoRA) are swapped in and out. In a standard Transformer setup, even a tiny change to this context forces the model to recompute the entire Key-Value (KV) cache from scratch. This is computationally wasteful and creates a bottleneck for real-time interaction. Existing systems focus on what to store in memory, but HeMeR addresses the neglected problem of how to efficiently serve that memory during inference.

Motivation of HeMeR: Naive recomputation vs. selective reconciliation.

Methods

The authors introduce a "memory-serving" framework that anchors memory contents as structured chunks. Instead of a flat string of text, the model sees a sequence of chunks, each with metadata like a persistent ID, semantic role, and associated KV range.

HeMeR employs two primary mechanisms to reconcile these chunks with the existing cache:

  1. Chunk-Level KV Reconciliation: When the agent's state changes, HeMeR compares the new memory context to the old one. If a memory chunk is simply moved to a different position in the prompt, HeMeR uses RoPE (Rotary Position Embedding) remapping to update the cached keys' positional phase without recomputing the values. If a chunk is new or its content has changed, only that specific range is recomputed.
  2. Adapter-Stable KV Reconciliation: Embodied agents often use different LoRA adapters for different tasks (e.g., one for "picking" and one for "cleaning"). Switching adapters usually invalidates the cache. HeMeR tracks which adapter was used to compute which KV state and selectively recomputes only the ranges affected by an adapter switch, preserving compatible states across the transition.
HeMeR overview showing how memory chunks are indexed and updated across steps.

Datasets

The researchers evaluated HeMeR across three distinct environments:

  • ALFWorld: A text-based household simulator where agents must solve multi-step tasks using a discrete command interface.
  • EmbodiedBench Habitat: A high-fidelity 3D simulator (using ReplicaCAD) where agents process RGB observations to perform tasks like "deposit apples on the TV stand."
  • Real-world Robotics: A physical setup featuring a UR7e robotic arm performing long-horizon tasks like "Make a vanilla Americano and a slice of toast" in a tabletop kitchen-café environment.

Key Insights and Ideas

  • Memory as a Serving Problem: The core "aha" moment is reframing heterogeneous memory use as a KV reconciliation problem. Efficiency isn't just about better retrieval; it's about how the model's internal state handles that retrieved data.
  • Position Independence via RoPE: The realization that we can "teleport" cached states to new positions in a prompt using simple mathematical rotations (RoPE remapping) is a powerful tool for dynamic contexts.
  • Cache Stability Equals Task Success: Interestingly, HeMeR doesn't just make robots faster; it makes them better. By preserving compatible KV states, the model maintains better internal consistency, leading to higher "active action ratios" (doing things that actually progress the task) and fewer redundant failures.
  • Localization of Drift: While naive cache reuse can lead to "hidden-state drift" (where errors accumulate over time), HeMeR's schema-guided approach confines these deviations to a small number of affected chunks, keeping the overall model representation stable.

Key Results

HeMeR consistently outperformed strong baselines like MemRL and LRAgent across all benchmarks:

  • ALFWorld: Achieved a 71.43% success rate (vs. 52.86% for MemRL) while slashing per-step latency from 0.90s to 0.18s.
  • Habitat: Reached a 76.27% success rate, significantly higher than the best cache-reuse baseline (KEEP at 59.40%).
  • Real-world: In physical robot trials, HeMeR achieved an 83.20% success rate and reduced per-step inference time from 2.94s to 1.87s.
Table 2: Performance on real-world embodied tasks. HEMER improves task success and reduces per-step inference time by preserving reusable KV states while using task-relevant memory contents.

Limitations and Future Work

The authors identify a few key areas for future exploration:

  • Multi-Agent Systems: The current work focuses on a single agent. In multi-agent environments, concurrent updates to shared memory would require new concurrency control mechanisms to prevent cache inconsistency.
  • Dependency on LoRA Design: The adapter-stable reconciliation relies on specific shared-attention LoRA designs. Extending this to more diverse adapter architectures remains an open challenge.
  • Scaling: While HeMeR scales well to 128 memory chunks, extremely long-horizon tasks with thousands of memory fragments may require more aggressive eviction or compression strategies.

Summary updated

Abstract

Embodied agents make decisions by combining information from heterogeneous memory modules, including declarative memory for explicit knowledge, procedural memory for learned routines, and working memory for current observations. During interaction, the memory contents required for decision making change with the agent's state, goal, and task progress, while many contents remain shared across consecutive decisions. However, existing memory modules are typically accessed through separate interfaces, so each updated memory context is often processed as a new input to the foundation model. This prevents selective key-value (KV) cache reuse for unchanged memory contents and forces full KV cache recomputation even when much of the memory context remains valid. We introduce HeMeR, an inference framework that performs KV reconciliation between dynamically updated memory contexts and reusable KV states. HeMeR tracks declarative, procedural, and working memory contents across action decisions, preserves KV states for unchanged contents, and reconciles only the KV states affected by memory-content updates. This enables efficient and stable inference as memory contents evolve during interaction. We evaluate HeMeR on ALFWorld, Habitat, and real-world robotic deployments requiring coordinated use of heterogeneous memory contents. In the real-world setting, HeMeR achieves 83.20% task success and reduces per-step inference time from 2.94 to 1.87 seconds compared with MemRL, while preserving compatible KV states as task progress updates the memory context.
BibTeX
@misc{chun2026hemer,
  author = {Saehun Chun and Wonje Choi and Jinwoo Jang and Taeyoon Kwack and Honguk Woo},
  title = {{HeMeR: Heterogeneous Memory Reconciliation for Embodied Agents via Structured KV Reuse}},
  year = {2026},
  eprint = {2611.16487},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16487},
}
Figure