TL;DR
ActMem is a novel Vision-Language-Action (VLA) framework that mimics human cognition by decoupling memory reasoning (deciding what to do based on history) from perceptual grounding (deciding how to act based on the current view). By using a "Latent Memory Chain-of-Thought" to condition a flow-matching action expert, it achieves state-of-the-art performance on long-horizon robotic tasks while maintaining high inference speeds.
What Problem Does This Paper Address?
In robotic manipulation, long-horizon tasks often require a "memory" of events that are no longer visible. For example, if a robot hides a block under one of three identical cups, the current visual frame doesn't tell it which cup to lift—it must remember the past interaction.
Existing memory-augmented VLA models generally fall into two traps:
- Retrieval-based models store past observations but struggle to reason about what those observations imply for the current step.
- Native-video models process the entire history in a single context window. However, as the history grows, the visual details needed for precise action (perception) compete for "attention" with the historical frames (memory), often leading to degraded performance or extreme computational latency.
The authors argue that memory and perception are distinct systems. Memory should reconstruct past experiences to determine the subtask, while perception should guide the motor execution in the present.
Methods
ActMem introduces an in-context memory framework built on a vision-language backbone (Qwen3-VL) and a flow-matching action expert. Its architecture is defined by three core components:
- Latent Memory Chain-of-Thought (Memory CoT): Instead of just passing a text string of the current subtask to the robot's "arms," ActMem uses the hidden states formed during subtask inference. These states act as a latent "Chain-of-Thought," carrying rich, high-dimensional information about task progress and spatial reasoning that text alone cannot capture.
- Perception Attention: To prevent history from "crowding out" the current view, the action expert uses gated cross-attention to look directly at the contextualized visual features of the current frame. This ensures the robot remains grounded in the immediate geometry of the scene.
- Efficient Memory CoT Inference: Since subtasks usually persist across several decision steps, ActMem reuses the previous subtask as a "draft." It verifies this draft in a single parallel pass rather than decoding token-by-token at every step, significantly reducing latency for long-history tasks.
Datasets
The model was evaluated on both simulated and real-world benchmarks:
- RoboMemArena: A simulation benchmark with 26 tasks across categories like "Transferring," "Occlusion," "Counting," and "Sequence."
- MIKASA-Robo: A benchmark focusing on history-dependent manipulation (e.g., Shell Game, RememberColor).
- Real-World Tasks: The authors designed four tasks to test different memory types:
- Exogenous: Cup-and-Block, Scissor Retrieval (events caused by external agents).
- Endogenous: Cabinet Search (tracking self-generated progress).
- Traceless: Tissue Counting (counting identical actions where the scene looks the same after each step).
Key Insights and Ideas
- The "Boundary" is Functional, Not Temporal: The paper posits that the split isn't between "past frames" and "present frames," but between "what must be inferred" (memory) and "what can be observed" (perception).
- Hidden States > Text: A key "aha" moment is that the hidden states of the subtask tokens contain more information than the decoded text. For a counting task, the hidden states clearly cluster by count (e.g., "first pour" vs "second pour"), even when the generated text is identical.
- Decoupling as a Scalability Solution: By separating the reasoning backbone from the action expert, the model can handle very long histories (up to 126 seconds/frames) without losing the fine-grained visual focus required for grasping and manipulation.
Key Results
ActMem consistently outperformed existing VLAs across all benchmarks:
- RoboMemArena: Achieved 71.1% task success, surpassing the strongest baseline (SimpleMemVLA) by 7.5 percentage points.
- MIKASA-Robo: Reached 73.8% average success, the highest among compared methods.
- Real-World: Achieved a 68.8% average success rate, which is 10.0 percentage points higher than the next best memory-augmented VLA.
- Efficiency: The "Efficient Inference" method reduced per-decision latency to 0.33–0.52s, making it 2.1x to 2.8x faster than standard inference on long histories.
Limitations and Future Work
While ActMem is efficient, it still attends over the full history in its KV cache. For tasks lasting tens of minutes or hours, this would eventually hit memory limits. The authors suggest that future work should involve selective memory retrieval in latent space—deciding which parts of the history are worth keeping rather than retaining every single frame. Additionally, the model currently uses fixed sampling rates for history, which might not be optimal for all task dynamics.
Summary updated