Abstract
Modern robotic foundation models typically process visual input as a single, holistic vector. While effective in controlled environments, these representations often struggle with scene distractors and environmental variations. This paper investigates whether an object-centric representation (OCR)—which decomposes a scene into individual "object slots"—provides a more robust and generalizable framework for robotic manipulation. Using a new benchmark, RoboShape, we demonstrate that OCR-based models significantly outperform holistic counterparts in complex tasks and generalize more effectively to unseen colors and object sizes.
Introduction
In the current landscape of robot learning, most vision-based policies treat the camera frame as a global entity. While this approach benefits from the power of large-scale pre-training, it lacks the structural inductive bias inherent in human perception: the ability to see the world as a collection of distinct, interacting entities. This holistic approach makes models vulnerable to "distractors"—irrelevant objects or background changes that alter the global feature vector without changing the task's underlying logic.
This research explores the hypothesis that object-centric representations are essential for generalizable robotics. By isolating objects into discrete slots, a model can theoretically ignore noise in the background or changes in irrelevant objects, focusing only on the entities necessary for the task at hand.
The RoboShape Benchmark
To rigorously test this hypothesis, the authors introduced RoboShape, a benchmark built on the ManiSkill simulation framework. RoboShape consists of 2,000 expert trajectories across three tasks of increasing complexity:
- Push Cube: Moving a red cube to a target location.
- Pick Cube: Grasping the red cube and lifting it to a specific coordinate.
- Place Cube in Bin: Picking the cube and successfully depositing it into a green bin.
To evaluate robustness, the benchmark includes three levels of out-of-distribution (OOD) scenarios:
- L1 (Novel Distractor Colors): Distractors in the scene have colors never encountered during training.
- L2 (Novel Backgrounds): The table or floor color is changed to an unseen hue.
- L3 (Novel Distractor Sizes): Distractors are scaled to randomized, unseen sizes.
Methodology
The study utilizes a two-stage framework combining unsupervised visual decomposition with a transformer-based policy.
1. Vision Pre-training (SAVi)
The model employs Slot-Attention for Video (SAVi) to learn a scene representation without human labels. By using an encoder-decoder architecture trained on a simple image reconstruction loss (MSE), the model learns to decompose video frames into a set of "slots." Each slot ideally represents a single object or a coherent part of the background. This temporal approach allows the model to maintain object identity even during partial occlusions, such as when the robot's arm blocks the target object.
2. Manipulation Learning
Once the vision backbone is frozen, the extracted slots are passed through a Transformer Observation Trunk. A learned [ACT] token aggregates information across slots and time. The final action is predicted by a Gaussian-Mixture Model (GMM) head, which outputs a 7D action vector (6D gripper pose and 1D gripper state).
Experimental Results
The OCR-based model (SAVi) was compared against two prominent holistic representations: DINO (a standard vision transformer) and R3M (a state-of-the-art robotic representation).
Task Performance and Complexity
On the simplest task (Pushing), holistic models like R3M performed well, likely due to their strong grasp of global dynamics. However, as task complexity increased, the gap widened significantly. In the Pick Cube task, SAVi achieved a 56% success rate, whereas both DINO and R3M failed completely (0% success). The holistic models were unable to perform the precise spatial reasoning required to grasp an object in the presence of distractors.
Generalization and Robustness
The most striking results appeared in the generalization tests. In the L1 scenario (unseen distractor colors), the holistic R3M model suffered a 60% drop in success rate. In contrast, the SAVi-based model only saw a 14% drop. Because the OCR model segments the scene, it can assign a new color to a "background" or "distractor" slot without affecting the representation of the "target cube" slot. This isolation provides a level of robustness that global vectors cannot match.
Discussion and Limitations
Despite the success in picking and pushing, all models—including the object-centric one—failed the Place in Bin task (0% success). Qualitative analysis suggests that the models struggled with depth perception from a single-view camera. The policy often perceived the cube as being inside the bin when it was actually hovering above or behind it.
Furthermore, while the model successfully separated the robot, target, and distractors into distinct channels without supervision, the reliance on reconstruction loss means the model might struggle in scenes with extremely complex textures or lighting where MSE is an insufficient training signal.
Conclusion
This study provides strong evidence that object-centric representations are a key ingredient for robust robotic manipulation. By decomposing scenes into entities, robots can better handle the unpredictability of real-world environments. Future work will focus on integrating multi-view setups (such as wrist-mounted cameras) to resolve depth ambiguity and scaling these representations to large-scale, real-world datasets like Open X-Embodiment.
Summary updated