A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning

Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, Xiaojuan Qi

TL;DR

This paper challenges the prevailing wisdom that Masked Autoencoders (MAE) trained on ego-centric data are the gold standard for robot vision. The authors introduce SlotMIM, a method that forces models to learn "object-centric" representations from diverse, uncurated data, significantly outperforming existing state-of-the-art models like MVP and VC-1 with much higher data efficiency.

What Problem Does This Paper Address?

Building a "visual cortex" for robots is a major hurdle in embodied AI. Currently, the field leans heavily on pre-training vision models using MAE on ego-centric video datasets (like Ego4D). However, these models often struggle to generalize across different robotic tasks, such as switching from fine-grained manipulation to wide-area navigation.

The authors identify a critical gap: while models like DINO and iBOT excel when trained on neatly cropped, single-object images (like ImageNet), their performance collapses when trained on "messy" real-world data (scene-centric or web-crawled). They discovered that this failure is directly linked to the models losing their ability to distinguish individual objects—a capability they call "objectness"—when the data becomes complex.

Methods

The authors propose SlotMIM, a framework designed to maintain object-centric learning even when the training data is not object-centric.

Overview of SlotMIM architecture and its core components.

Key innovations include:

  • Representation Bottleneck: Unlike standard models that use thousands of visual "prototypes," SlotMIM uses a much smaller set (e.g., 512). This creates a bottleneck that forces the model to group image patches into meaningful, high-level "slots" (objects) rather than getting lost in fine-grained textures.
  • Cross-View Consistency: The model is trained to ensure that the same object "slot" is identified consistently across different augmented views of the same scene.
  • Object-Level Contrastive Learning: Instead of just comparing individual pixels or patches, SlotMIM performs contrastive learning at the level of these discovered "slots," making the learned representations more discriminative for downstream tasks.

Datasets

The study utilizes a diverse array of datasets to test generalization:

  • Pre-training: ImageNet (Object-centric), COCO+ (Scene-centric), CC12M (Web-crawled), and Ego4D (Ego-centric).
  • Scaling: The authors introduced DetSoup, a 4-million image scene-centric dataset combining COCO, OpenImages, Objects365, and LVIS.
  • Evaluation: The models were tested on a comprehensive benchmark including:
  • Visuomotor Control: Franka Kitchen and Meta-World.
  • Navigation: ObjectNav (in HM3D) and ImageNav (in Gibson).
  • Perception: Pascal VOC and ADE20K segmentation, plus COCO detection.

Key Insights and Ideas

  • Objectness Predicts Success: The authors found a strong correlation ($r=0.72$) between a model's internal ability to segment objects (without explicit labels) and its success rate in robotic manipulation tasks.
  • The Inverse Scaling Trap: A surprising finding was that for standard DINO and iBOT models, more data isn't always better. When scaled up on scene-centric data, these models actually lost performance on manipulation tasks due to "over-compression" of low-level visual details. SlotMIM avoids this trap.
  • Data-Task Alignment: There is no single "best" dataset. Ego-centric data is superior for manipulation (hand-eye coordination), while scene-centric data is far better for navigation (understanding layouts).

Key Results

SlotMIM demonstrates remarkable data efficiency and raw performance:

  • Efficiency: SlotMIM pre-trained on just 241K samples outperformed prior state-of-the-art models (like MVP and VC-1) that were trained on over 1 million samples.
  • Manipulation: On the Meta-World benchmark, SlotMIM achieved success rates near 80%, while standard iBOT and DINO models trailed significantly when trained on the same non-object-centric data.
  • Navigation: When trained on the "DetSoup" dataset, SlotMIM improved ObjectNav performance by 6.6% over previous best-in-class models.
Figure 1. An overview of this paper. (a) We conduct a comprehensive study evaluating pre-trained vision models (PVMs) on visuomotor control and perception tasks, analyzing how different pretraining (model, data) combinations affect performance. Our analysis reveals that DINO/iBOT excels while MAE underperforms. (b) We investigate the performance drop of DINO/iBOT when trained on non-(single- )object-centric (NOC) data, discovering they struggle to learn objectness from NOC data—a capability that strongly correlates with robot manipulation performance. (c) We introduce SlotMIM, which incorporates explicit objectness guidance during training to effectively learn object-centric representations from NOC data. (d) Through scaled-up pre-training and evaluation across six tasks, we demonstrate that SlotMIM adaptively learns different types of objectness based on the pre-training dataset characteristics, outperforming existing methods.

Limitations and Future Work

While SlotMIM shows great promise, the authors note several areas for further exploration:

  • Computational Cost: Evaluating these models across diverse robotic benchmarks is extremely GPU-intensive, limiting the number of hyperparameter variations that could be tested.
  • Task Complexity: The current evaluation focuses on behavior cloning and basic navigation. Future work is needed to see how SlotMIM handles language-conditioned tasks or complex, long-horizon real-world robotic deployments.
  • Foundation Model Integration: Exploring how SlotMIM can be integrated into larger generative "Vision-Language-Action" (VLA) models is a promising next step.

Summary updated

Abstract

Pre-trained vision models (PVMs) are fundamental to modern robotics, yet their optimal configuration remains unclear. Through systematic evaluation, we find that while DINO and iBOT outperform MAE across visuomotor control and perception tasks, they struggle when trained on non-(single-)object-centric (NOC) data--a limitation strongly correlated with their diminished ability to learn object-centric representations. This investigation indicates that the ability to form object-centric representations from the non-object-centric robotics dataset is the key to success for PVMs. Motivated by this discovery, we designed SlotMIM, a method that induces object-centric representations by introducing a semantic bottleneck to reduce the number of prototypes to encourage the emergence of objectness as well as cross-view consistency regularization for encouraging multiview invariance. Our experiments encompass pre-training on object-centric, scene-centric, web-crawled, and ego-centric data. Across all settings, our approach learns transferrable representations and achieves significant improvements over prior work in image recognition, scene understanding, and robot learning evaluations. When scaled up with million-scale datasets, our method also demonstrates superior data efficiency and scalability. Our code and models are publicly available at https://github.com/CVMI-Lab/SlotMIM.
BibTeX
@inproceedings{wen2025data,
  author = {Xin Wen and Bingchen Zhao and Yilun Chen and Jiangmiao Pang and Xiaojuan Qi},
  title = {{A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01134},
}
Figure