ActMem: From Latent Memory Reasoning to Grounded Robotic Action

Linfeng Dong, Jianwei Niu, Jiuyi Zhao, Ruibing Feng, Xinghao Wu, Wu Yang, Shuai Tao, Wulong Liu, Mu LiarXiv 2026 · arXiv

TL;DR

ActMem is a novel Vision-Language-Action (VLA) framework that mimics human cognition by decoupling memory reasoning (deciding what to do based on history) from perceptual grounding (deciding how to act based on the current view). By using a "Latent Memory Chain-of-Thought" to condition a flow-matching action expert, it achieves state-of-the-art performance on long-horizon robotic tasks while maintaining high inference speeds.

What Problem Does This Paper Address?

In robotic manipulation, long-horizon tasks often require a "memory" of events that are no longer visible. For example, if a robot hides a block under one of three identical cups, the current visual frame doesn't tell it which cup to lift—it must remember the past interaction.

Existing memory-augmented VLA models generally fall into two traps:

  1. Retrieval-based models store past observations but struggle to reason about what those observations imply for the current step.
  2. Native-video models process the entire history in a single context window. However, as the history grows, the visual details needed for precise action (perception) compete for "attention" with the historical frames (memory), often leading to degraded performance or extreme computational latency.

The authors argue that memory and perception are distinct systems. Memory should reconstruct past experiences to determine the subtask, while perception should guide the motor execution in the present.

Human-inspired vs. ActMem approach

Methods

ActMem introduces an in-context memory framework built on a vision-language backbone (Qwen3-VL) and a flow-matching action expert. Its architecture is defined by three core components:

  • Latent Memory Chain-of-Thought (Memory CoT): Instead of just passing a text string of the current subtask to the robot's "arms," ActMem uses the hidden states formed during subtask inference. These states act as a latent "Chain-of-Thought," carrying rich, high-dimensional information about task progress and spatial reasoning that text alone cannot capture.
  • Perception Attention: To prevent history from "crowding out" the current view, the action expert uses gated cross-attention to look directly at the contextualized visual features of the current frame. This ensures the robot remains grounded in the immediate geometry of the scene.
  • Efficient Memory CoT Inference: Since subtasks usually persist across several decision steps, ActMem reuses the previous subtask as a "draft." It verifies this draft in a single parallel pass rather than decoding token-by-token at every step, significantly reducing latency for long-history tasks.
Figure 2. Overview of ActMem. (a) A pretrained vision-language backbone (Qwen3-VL) encodes the instruction, the timestamped historical frames, and the current perception, and performs latent memory reasoning inside the backbone to produce the subtask together with its Latent Memory CoT states Ct, which condition a flow-matching action expert; Perception Attention additionally routes the current visual features to the action expert. (b) Inside the backbone, the subtask tokens that form Memory CoT follow the history and the current view in the sequence, so under the causal mask they read both, and their hidden states become Ct; the action expert receives only Ct, not the history tokens. (c) Efficient Memory CoT Inference. Consecutive decisions usually share the same subtask, so the previous subtask is reused as a draft and verified against the updated context in a single parallel pass. If it is accepted, its hidden states are recomputed to give the new Ct, with no token-by-token decoding.

Datasets

The model was evaluated on both simulated and real-world benchmarks:

  • RoboMemArena: A simulation benchmark with 26 tasks across categories like "Transferring," "Occlusion," "Counting," and "Sequence."
  • MIKASA-Robo: A benchmark focusing on history-dependent manipulation (e.g., Shell Game, RememberColor).
  • Real-World Tasks: The authors designed four tasks to test different memory types:
  • Exogenous: Cup-and-Block, Scissor Retrieval (events caused by external agents).
  • Endogenous: Cabinet Search (tracking self-generated progress).
  • Traceless: Tissue Counting (counting identical actions where the scene looks the same after each step).

Key Insights and Ideas

  • The "Boundary" is Functional, Not Temporal: The paper posits that the split isn't between "past frames" and "present frames," but between "what must be inferred" (memory) and "what can be observed" (perception).
  • Hidden States > Text: A key "aha" moment is that the hidden states of the subtask tokens contain more information than the decoded text. For a counting task, the hidden states clearly cluster by count (e.g., "first pour" vs "second pour"), even when the generated text is identical.
  • Decoupling as a Scalability Solution: By separating the reasoning backbone from the action expert, the model can handle very long histories (up to 126 seconds/frames) without losing the fine-grained visual focus required for grasping and manipulation.
Visualization of Latent Memory CoT encoding

Key Results

ActMem consistently outperformed existing VLAs across all benchmarks:

  • RoboMemArena: Achieved 71.1% task success, surpassing the strongest baseline (SimpleMemVLA) by 7.5 percentage points.
  • MIKASA-Robo: Reached 73.8% average success, the highest among compared methods.
  • Real-World: Achieved a 68.8% average success rate, which is 10.0 percentage points higher than the next best memory-augmented VLA.
  • Efficiency: The "Efficient Inference" method reduced per-decision latency to 0.33–0.52s, making it 2.1x to 2.8x faster than standard inference on long histories.
Real-world performance comparison

Limitations and Future Work

While ActMem is efficient, it still attends over the full history in its KV cache. For tasks lasting tens of minutes or hours, this would eventually hit memory limits. The authors suggest that future work should involve selective memory retrieval in latent space—deciding which parts of the history are worth keeping rather than retaining every single frame. Additionally, the model currently uses fixed sampling rates for history, which might not be optimal for all task dynamics.

Summary updated

Abstract

Vision-Language-Action (VLA) models have shown promising results in real-world long-horizon manipulation, where memory plays a crucial role. However, existing memory-augmented VLAs either retrieve past observations without reasoning about what they imply, or reason over the full history in a shared context where it crowds out current perception. Cognitive science treats them as distinct systems: memory reconstructs past experience to decide what to do, while perception guides how to act. We therefore propose ActMem, an in-context memory framework that decouples memory reasoning from perceptual grounding within a single vision-language backbone. Built on a vision-language model with native video understanding, ActMem infers the current subtask from a timestamped interaction history and keeps the hidden states formed during this inference as a Latent Memory Chain-of-Thought (Memory CoT), which conditions a flow-matching action expert. Perception Attention further lets the expert attend directly to the current view. To keep inference fast under long histories, Efficient Memory CoT Inference reuses computation across consecutive decisions without changing the output. ActMem achieves 71.1% task success and 80.8% completion success on RoboMemArena, surpassing the strongest baseline by 7.5 and 8.7 percentage points, and 73.8% average success on MIKASA-Robo, the best among all compared methods. In the real world, it reaches 68.8% average success on exogenous, endogenous, and traceless memory tasks, 10.0 percentage points above the strongest memory-augmented VLA. Code and checkpoints will be released at https://github.com/LigphiDonk/ActMem and https://huggingface.co/donk111/ActMem.
BibTeX
@misc{dong2026actmem,
  author = {Linfeng Dong and Jianwei Niu and Jiuyi Zhao and Ruibing Feng and Xinghao Wu and Wu Yang and Shuai Tao and Wulong Liu and Mu Li},
  title = {{ActMem: From Latent Memory Reasoning to Grounded Robotic Action}},
  year = {2026},
  eprint = {2611.25461},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25461},
}
Figure