Self-Supervised Learning of Object Parts for Semantic Segmentation

Adrian Ziegler, Yuki M. Asano

Abstract

Progress in self-supervised learning (SSL) has yielded powerful image representation methods, primarily at the image level. However, dense tasks like unsupervised semantic segmentation have lagged behind due to the challenge of defining object categories without labels. This paper introduces Leopart, a framework that shifts the focus to learning object parts. By leveraging Vision Transformers (ViTs) and a spatially dense clustering task, Leopart learns representations that are versatile across various object definitions. Our method surpasses state-of-the-art benchmarks by 3% to 17% mIoU. Furthermore, we demonstrate that a community-detection-based approach for merging discovered parts enables robust fully unsupervised segmentation.

Introduction

Self-supervised learning has transformed computer vision, yet its success has been largely confined to image-level classification. Dense, pixel-level tasks like semantic segmentation remain difficult because they require spatially diverse representations. Standard SSL methods often struggle with the "object definition" problem: determining what constitutes an object (e.g., is a car one entity or a collection of parts?) without human guidance.

Leopart (Learning Object Parts) addresses this by focusing on constituent parts rather than whole objects. Parts, such as wheels or legs, are more consistent across datasets and can be grouped a posteriori to form objects. This approach allows the model to understand scenes more flexibly without relying on pre-defined labels.

Methodology: The Leopart Framework

Leopart utilizes a dense clustering pretext task to train Vision Transformers. The core architecture follows a teacher-student setup inspired by the SwAV algorithm but adapted for spatial tokens.

Overview of the Leopart training framework, showing the teacher-student setup for dense patch clustering.

The process involves four key components:

  1. Dense Clustering with ViTs: The model generates embeddings for every 16x16 patch rather than a single global embedding.
  2. Swapped Prediction Task: A teacher network processes a global view and assigns patches to clusters via Sinkhorn-Knopp clustering. A student network, viewing a different crop, must predict these cluster assignments.
  3. Foreground Focusing: To prioritize relevant features, the model uses the ViT’s internal self-attention maps to weight the loss, effectively focusing on foreground objects.
  4. Unsupervised Merging via Community Detection: For fully unsupervised scenarios, Leopart builds a co-occurrence network of parts. It then applies the Infomap algorithm—a community detection tool from network science—to group these parts into coherent object categories based on their statistical proximity.

Experimental Evaluation

The model was evaluated on major benchmarks including Pascal VOC 2012, COCO-Stuff, and ADE20k. Leopart consistently outperformed previous self-supervised methods like DINO, MoCov3, and MaskContrast.

Leopart (CBFE+CD) achieves superior performance in both foreground extraction and semantic segmentation compared to existing SSL backbones.

Key Results

  • State-of-the-Art Performance: Leopart improved mIoU by 3% to 17% across Pascal VOC and COCO datasets.
  • Closing the Supervised Gap: On Pascal VOC, Leopart reached 69.3 mIoU, nearly matching the 68.1 mIoU of a supervised ViT.
  • Fully Unsupervised Success: In the absence of any labels even at test time, the community detection method achieved 41.7 mIoU, significantly higher than the previous best of 35.0 mIoU.
  • Fine-Grained Part Discovery: On the ADE20k dataset, the model successfully identified 111 distinct part classes, demonstrating high-resolution feature discrimination.

Discussion and Conclusion

The success of Leopart suggests that object parts are a "universal language" for scene understanding. While object definitions are often task-specific, parts like "leaves" or "wheels" are consistent across domains. By combining the localization strengths of ViTs with dense clustering, Leopart provides a more transferable representation for downstream tasks.

However, limitations remain. The 16x16 patch resolution can miss fine-grained boundaries, and the current "hard" assignment of parts to objects doesn't account for shared components (e.g., a wheel appearing on both cars and buses). Future work will explore pixel-level learning and "soft" assignments to further bridge the gap toward human-level unsupervised perception.

Summary updated

Abstract

Progress in self-supervised learning has brought strong image representation learning methods. Yet so far, it has mostly focused on image-level learning. In turn, tasks such as unsupervised image segmentation have not benefited from this trend as they require spatially-diverse representations. However, learning dense representations is challenging, as in the unsupervised context it is not clear how to guide the model to learn representations that correspond to various potential object categories. In this paper, we argue that self-supervised learning of object parts is a solution to this issue. Object parts are generalizable: they are a priori independent of an object definition, but can be grouped to form objects a posteriori. To this end, we leverage the recently proposed Vision Transformer's capability of attending to objects and combine it with a spatially dense clustering task for fine-tuning the spatial tokens. Our method surpasses the state-of-the-art on three semantic segmentation benchmarks by 17%-3%, showing that our representations are versatile under various object definitions. Finally, we extend this to fully unsupervised segmentation - which refrains completely from using label information even at test-time - and demonstrate that a simple method for automatically merging discovered object parts based on community detection yields substantial gains..
BibTeX
@inproceedings{ziegler2022self,
  author = {Adrian Ziegler and Yuki M. Asano},
  title = {{Self-Supervised Learning of Object Parts for Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=D2fZSYXeH0h},
}
Figure