TL;DR
Most modern robotic policies fail when a task requires remembering past interactions with visually identical objects. This paper introduces LIBERO-Mem, a new benchmark specifically designed to stress-test long-horizon, object-centric memory in non-Markovian settings, alongside Embodied-SlotSSM, a scalable framework that tracks individual object states over time using slot-based state-space models.
What Problem Does This Paper Address?
Standard Vision-Language-Action (VLA) models operate under a reactive, Markovian assumption: they predict the next robotic action based solely on the current camera frame and the language instruction. However, real-world tasks like cooking, organizing, or assembly are inherently Partially Observable Markov Decision Processes (POMDPs).
Consider an instruction like "pick up the bowl twice." To an agent without memory, the visual scene before the first pick-up looks identical to the scene before the second pick-up. Without tracking history, a robot can get stuck in infinite loops, repeat completed actions, or miss steps entirely. While transformers can handle history by processing sequences of past frames, doing so introduces a massive computational burden, as visual tokens scale linearly with the number of frames, making long-horizon tracking intractable.
Methods
The authors introduce Embodied-SlotSSM, a framework designed to maintain a persistent, object-centric memory without the token explosion typical of standard transformers.
The architecture consists of three main pillars:
- Slot Attention for Object Localization: Instead of treating a frame as a dense grid of pixels, the model uses Slot Attention to decompose the scene into a fixed set of discrete, persistent latent "slots" representing individual objects.
- Slot State-Space Modeling (SlotSSM): To track these objects across time efficiently, the model applies a State-Space Model (SSM) to each individual slot. Instead of predicting just the next step, it predicts a localized temporal window of both past and future states, which enforces temporal consistency and smooths out noisy transitions.
- Relation Encoder & Action Decoding: A lightweight relation encoder performs cross-attention between the slot memories and raw visual features to capture inter-object relationships. These tokens are then fed into an LLM action decoder to output precise control commands ($\Delta x, \Delta \theta, \Delta \text{grasp}$).
Datasets
The models were evaluated on two benchmark suites:
- LIBERO-Goal: An existing benchmark consisting of standard, Markovian manipulation tasks used to establish baseline visuomotor performance.
- LIBERO-Mem (New): A novel benchmark introduced in this paper containing 10 tasks designed to test four distinct object-centric memory dimensions:
- Object Motion (OM): Recalling the immediate prior action (e.g., whether the object was just lifted or placed).
- Object Sequence (OS): Tracking repetitive actions (e.g., repeating a placement 3, 5, or 7 times).
- Multi-Object Sequence (OR): Tracking the relative temporal order of multiple objects (e.g., swapping bowls from left to right).
- Multi-Object Occlusion (OO): Relying on memory to identify targets that are hidden from view (e.g., finding a bowl hidden inside a basket).
Key Insights and Ideas
- History Resolves Visual Ambiguity: When objects are visually indistinguishable, their unique interaction histories are the only features that differentiate them. Object-specific tracking is mandatory for non-Markovian manipulation.
- Overcoming the Token Explosion: By compressing dense image frames into a small number of object slots (e.g., 16) and passing them through an SSM instead of full temporal attention, the model decouples memory capacity from sequence length.
- Bidirectional Windowing as a Supervisor: Forcing the SlotSSM to reconstruct both past and future object states during training provides a powerful inductive bias that stabilizes object tracking, even through brief occlusions.
Key Results
- General Tasks (LIBERO-Goal): An oracle-supported implementation of the proposed method (Naive E-SlotSSM) achieved an 83.0% average success rate, outperforming prior slot-based models like SlotVLA (75.5%) and standard single-frame reactive baselines (32%).
- Memory-Intensive Tasks (LIBERO-Mem): Standard reactive models ($\pi_0$) and SlotVLA struggled heavily on long-horizon tasks, averaging a mere 5.0% subgoal completion rate. Naive E-SlotSSM significantly improved this to 14.8% average subgoal completion, showing clear success in tracking multi-step repetitions (e.g., completing 33.3% of the 3x repeated bowl placement task) and managing occlusions.
Limitations and Future Work
While Embodied-SlotSSM marks a significant step forward, the authors note that the empirical version evaluated on LIBERO-Mem (Naive E-SlotSSM) relies on an oracle subgoal monitor—meaning it receives text-embedded subgoal hints rather than inferring them completely autonomously.
Additionally, the benchmark is currently restricted to simulation. Future work will focus on transitioning the framework to physical hardware and developing methods for self-discovered subgoal reasoning under object-level partial observability.
Summary updated