TL;DR
This paper challenges the prevailing wisdom that Masked Autoencoders (MAE) trained on ego-centric data are the gold standard for robot vision. The authors introduce SlotMIM, a method that forces models to learn "object-centric" representations from diverse, uncurated data, significantly outperforming existing state-of-the-art models like MVP and VC-1 with much higher data efficiency.
What Problem Does This Paper Address?
Building a "visual cortex" for robots is a major hurdle in embodied AI. Currently, the field leans heavily on pre-training vision models using MAE on ego-centric video datasets (like Ego4D). However, these models often struggle to generalize across different robotic tasks, such as switching from fine-grained manipulation to wide-area navigation.
The authors identify a critical gap: while models like DINO and iBOT excel when trained on neatly cropped, single-object images (like ImageNet), their performance collapses when trained on "messy" real-world data (scene-centric or web-crawled). They discovered that this failure is directly linked to the models losing their ability to distinguish individual objects—a capability they call "objectness"—when the data becomes complex.
Methods
The authors propose SlotMIM, a framework designed to maintain object-centric learning even when the training data is not object-centric.
Key innovations include:
- Representation Bottleneck: Unlike standard models that use thousands of visual "prototypes," SlotMIM uses a much smaller set (e.g., 512). This creates a bottleneck that forces the model to group image patches into meaningful, high-level "slots" (objects) rather than getting lost in fine-grained textures.
- Cross-View Consistency: The model is trained to ensure that the same object "slot" is identified consistently across different augmented views of the same scene.
- Object-Level Contrastive Learning: Instead of just comparing individual pixels or patches, SlotMIM performs contrastive learning at the level of these discovered "slots," making the learned representations more discriminative for downstream tasks.
Datasets
The study utilizes a diverse array of datasets to test generalization:
- Pre-training: ImageNet (Object-centric), COCO+ (Scene-centric), CC12M (Web-crawled), and Ego4D (Ego-centric).
- Scaling: The authors introduced DetSoup, a 4-million image scene-centric dataset combining COCO, OpenImages, Objects365, and LVIS.
- Evaluation: The models were tested on a comprehensive benchmark including:
- Visuomotor Control: Franka Kitchen and Meta-World.
- Navigation: ObjectNav (in HM3D) and ImageNav (in Gibson).
- Perception: Pascal VOC and ADE20K segmentation, plus COCO detection.
Key Insights and Ideas
- Objectness Predicts Success: The authors found a strong correlation ($r=0.72$) between a model's internal ability to segment objects (without explicit labels) and its success rate in robotic manipulation tasks.
- The Inverse Scaling Trap: A surprising finding was that for standard DINO and iBOT models, more data isn't always better. When scaled up on scene-centric data, these models actually lost performance on manipulation tasks due to "over-compression" of low-level visual details. SlotMIM avoids this trap.
- Data-Task Alignment: There is no single "best" dataset. Ego-centric data is superior for manipulation (hand-eye coordination), while scene-centric data is far better for navigation (understanding layouts).
Key Results
SlotMIM demonstrates remarkable data efficiency and raw performance:
- Efficiency: SlotMIM pre-trained on just 241K samples outperformed prior state-of-the-art models (like MVP and VC-1) that were trained on over 1 million samples.
- Manipulation: On the Meta-World benchmark, SlotMIM achieved success rates near 80%, while standard iBOT and DINO models trailed significantly when trained on the same non-object-centric data.
- Navigation: When trained on the "DetSoup" dataset, SlotMIM improved ObjectNav performance by 6.6% over previous best-in-class models.
Limitations and Future Work
While SlotMIM shows great promise, the authors note several areas for further exploration:
- Computational Cost: Evaluating these models across diverse robotic benchmarks is extremely GPU-intensive, limiting the number of hyperparameter variations that could be tested.
- Task Complexity: The current evaluation focuses on behavior cloning and basic navigation. Future work is needed to see how SlotMIM handles language-conditioned tasks or complex, long-horizon real-world robotic deployments.
- Foundation Model Integration: Exploring how SlotMIM can be integrated into larger generative "Vision-Language-Action" (VLA) models is a promising next step.
Summary updated