Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective

Nhat Chung, Taisei Hanyu, Toan Nguyen, Huy Le, Frederick Bumgarner, Duy Minh Ho Nguyen, Khoa Vo, Kashu Yamazaki, Chase Rainwater, Tung Kieu, Anh Nguyen, Ngan Le

TL;DR

Most modern robotic policies fail when a task requires remembering past interactions with visually identical objects. This paper introduces LIBERO-Mem, a new benchmark specifically designed to stress-test long-horizon, object-centric memory in non-Markovian settings, alongside Embodied-SlotSSM, a scalable framework that tracks individual object states over time using slot-based state-space models.

What Problem Does This Paper Address?

Standard Vision-Language-Action (VLA) models operate under a reactive, Markovian assumption: they predict the next robotic action based solely on the current camera frame and the language instruction. However, real-world tasks like cooking, organizing, or assembly are inherently Partially Observable Markov Decision Processes (POMDPs).

Consider an instruction like "pick up the bowl twice." To an agent without memory, the visual scene before the first pick-up looks identical to the scene before the second pick-up. Without tracking history, a robot can get stuck in infinite loops, repeat completed actions, or miss steps entirely. While transformers can handle history by processing sequences of past frames, doing so introduces a massive computational burden, as visual tokens scale linearly with the number of frames, making long-horizon tracking intractable.

Methods

The authors introduce Embodied-SlotSSM, a framework designed to maintain a persistent, object-centric memory without the token explosion typical of standard transformers.

The architecture consists of three main pillars:

  1. Slot Attention for Object Localization: Instead of treating a frame as a dense grid of pixels, the model uses Slot Attention to decompose the scene into a fixed set of discrete, persistent latent "slots" representing individual objects.
  2. Slot State-Space Modeling (SlotSSM): To track these objects across time efficiently, the model applies a State-Space Model (SSM) to each individual slot. Instead of predicting just the next step, it predicts a localized temporal window of both past and future states, which enforces temporal consistency and smooths out noisy transitions.
  3. Relation Encoder & Action Decoding: A lightweight relation encoder performs cross-attention between the slot memories and raw visual features to capture inter-object relationships. These tokens are then fed into an LLM action decoder to output precise control commands ($\Delta x, \Delta \theta, \Delta \text{grasp}$).
Embodied-SlotSSM Framework

Datasets

The models were evaluated on two benchmark suites:

  • LIBERO-Goal: An existing benchmark consisting of standard, Markovian manipulation tasks used to establish baseline visuomotor performance.
  • LIBERO-Mem (New): A novel benchmark introduced in this paper containing 10 tasks designed to test four distinct object-centric memory dimensions:
  • Object Motion (OM): Recalling the immediate prior action (e.g., whether the object was just lifted or placed).
  • Object Sequence (OS): Tracking repetitive actions (e.g., repeating a placement 3, 5, or 7 times).
  • Multi-Object Sequence (OR): Tracking the relative temporal order of multiple objects (e.g., swapping bowls from left to right).
  • Multi-Object Occlusion (OO): Relying on memory to identify targets that are hidden from view (e.g., finding a bowl hidden inside a basket).
Figure 1: LIBERO-Mem: robotic manipulation tasks of object-level POMDP dependencies. These tasks require memory of prior actions and object-specific state tracking beyond what purely Markovian or fully observable policies can handle, highlighting the importance of persistent, object-specific memory for short- and long-horizon reasoning across visually similar inputs. In (a) object motion (OM), the robot must recall its last action (e.g., pick up or place down) to act correctly. In (b) object sequence (OS), success depends on remembering how many times an object has been manipulated, since visual cues are insufficient. In (c) multi-object sequence (OR), the robot must track the temporal order of object relations and interactions (e.g., from left to right). In (d) multi-object occlusion (OO), occluded objects require the robot to rely on memory of past placements to identify targets.

Key Insights and Ideas

  • History Resolves Visual Ambiguity: When objects are visually indistinguishable, their unique interaction histories are the only features that differentiate them. Object-specific tracking is mandatory for non-Markovian manipulation.
  • Overcoming the Token Explosion: By compressing dense image frames into a small number of object slots (e.g., 16) and passing them through an SSM instead of full temporal attention, the model decouples memory capacity from sequence length.
  • Bidirectional Windowing as a Supervisor: Forcing the SlotSSM to reconstruct both past and future object states during training provides a powerful inductive bias that stabilizes object tracking, even through brief occlusions.

Key Results

  • General Tasks (LIBERO-Goal): An oracle-supported implementation of the proposed method (Naive E-SlotSSM) achieved an 83.0% average success rate, outperforming prior slot-based models like SlotVLA (75.5%) and standard single-frame reactive baselines (32%).
  • Memory-Intensive Tasks (LIBERO-Mem): Standard reactive models ($\pi_0$) and SlotVLA struggled heavily on long-horizon tasks, averaging a mere 5.0% subgoal completion rate. Naive E-SlotSSM significantly improved this to 14.8% average subgoal completion, showing clear success in tracking multi-step repetitions (e.g., completing 33.3% of the 3x repeated bowl placement task) and managing occlusions.

Limitations and Future Work

While Embodied-SlotSSM marks a significant step forward, the authors note that the empirical version evaluated on LIBERO-Mem (Naive E-SlotSSM) relies on an oracle subgoal monitor—meaning it receives text-embedded subgoal hints rather than inferring them completely autonomously.

Additionally, the benchmark is currently restricted to simulation. Future work will focus on transitioning the framework to physical hardware and developing methods for self-discovered subgoal reasoning under object-level partial observability.

Summary updated

Abstract

As embodied agents operate in increasingly complex environments, the ability to perceive, track, and reason about individual object instances over time becomes essential, especially in tasks requiring sequenced interactions with visually similar objects. In these non-Markovian settings, key decision cues are often hidden in object-specific histories rather than the current scene. Without persistent memory of prior interactions (what has been interacted with, where it has been, or how it has changed) visuomotor policies may fail, repeat past actions, or overlook completed ones. To surface this challenge, we introduce LIBERO-Mem, a non-Markovian task suite for stress-testing robotic manipulation under object-level partial observability. It combines short- and long-horizon object tracking with temporally sequenced subgoals, requiring reasoning beyond the current frame. However, vision-language-action (VLA) models often struggle in such settings, with token scaling quickly becoming intractable even for tasks spanning just a few hundred frames. We propose Embodied-SlotSSM, a slot-centric VLA framework built for temporal scalability. It maintains spatio-temporally consistent slot identities and leverages them through two mechanisms: (1) slot-state-space modeling for reconstructing short-term history, and (2) a relational encoder to align the input tokens with action decoding. Together, these components enable temporally grounded, context-aware action prediction. Experiments show Embodied-SlotSSM's baseline performance on LIBERO-Mem and general tasks, offering a scalable solution for non-Markovian reasoning in object-centric robotic policies.
BibTeX
@inproceedings{chung2026rethinking,
  author = {Nhat Chung and Taisei Hanyu and Toan Nguyen and Huy Le and Frederick Bumgarner and Duy Minh Ho Nguyen and Khoa Vo and Kashu Yamazaki and Chase Rainwater and Tung Kieu and Anh Nguyen and Ngan Le},
  title = {{Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspective}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i5.37337},
}
Figure