Abstract
Progress in self-supervised learning (SSL) has yielded powerful image representation methods, primarily at the image level. However, dense tasks like unsupervised semantic segmentation have lagged behind due to the challenge of defining object categories without labels. This paper introduces Leopart, a framework that shifts the focus to learning object parts. By leveraging Vision Transformers (ViTs) and a spatially dense clustering task, Leopart learns representations that are versatile across various object definitions. Our method surpasses state-of-the-art benchmarks by 3% to 17% mIoU. Furthermore, we demonstrate that a community-detection-based approach for merging discovered parts enables robust fully unsupervised segmentation.
Introduction
Self-supervised learning has transformed computer vision, yet its success has been largely confined to image-level classification. Dense, pixel-level tasks like semantic segmentation remain difficult because they require spatially diverse representations. Standard SSL methods often struggle with the "object definition" problem: determining what constitutes an object (e.g., is a car one entity or a collection of parts?) without human guidance.
Leopart (Learning Object Parts) addresses this by focusing on constituent parts rather than whole objects. Parts, such as wheels or legs, are more consistent across datasets and can be grouped a posteriori to form objects. This approach allows the model to understand scenes more flexibly without relying on pre-defined labels.
Methodology: The Leopart Framework
Leopart utilizes a dense clustering pretext task to train Vision Transformers. The core architecture follows a teacher-student setup inspired by the SwAV algorithm but adapted for spatial tokens.
The process involves four key components:
- Dense Clustering with ViTs: The model generates embeddings for every 16x16 patch rather than a single global embedding.
- Swapped Prediction Task: A teacher network processes a global view and assigns patches to clusters via Sinkhorn-Knopp clustering. A student network, viewing a different crop, must predict these cluster assignments.
- Foreground Focusing: To prioritize relevant features, the model uses the ViT’s internal self-attention maps to weight the loss, effectively focusing on foreground objects.
- Unsupervised Merging via Community Detection: For fully unsupervised scenarios, Leopart builds a co-occurrence network of parts. It then applies the Infomap algorithm—a community detection tool from network science—to group these parts into coherent object categories based on their statistical proximity.
Experimental Evaluation
The model was evaluated on major benchmarks including Pascal VOC 2012, COCO-Stuff, and ADE20k. Leopart consistently outperformed previous self-supervised methods like DINO, MoCov3, and MaskContrast.
Key Results
- State-of-the-Art Performance: Leopart improved mIoU by 3% to 17% across Pascal VOC and COCO datasets.
- Closing the Supervised Gap: On Pascal VOC, Leopart reached 69.3 mIoU, nearly matching the 68.1 mIoU of a supervised ViT.
- Fully Unsupervised Success: In the absence of any labels even at test time, the community detection method achieved 41.7 mIoU, significantly higher than the previous best of 35.0 mIoU.
- Fine-Grained Part Discovery: On the ADE20k dataset, the model successfully identified 111 distinct part classes, demonstrating high-resolution feature discrimination.
Discussion and Conclusion
The success of Leopart suggests that object parts are a "universal language" for scene understanding. While object definitions are often task-specific, parts like "leaves" or "wheels" are consistent across domains. By combining the localization strengths of ViTs with dense clustering, Leopart provides a more transferable representation for downstream tasks.
However, limitations remain. The 16x16 patch resolution can miss fine-grained boundaries, and the current "hard" assignment of parts to objects doesn't account for shared components (e.g., a wheel appearing on both cars and buses). Future work will explore pixel-level learning and "soft" assignments to further bridge the gap toward human-level unsupervised perception.
Summary updated