Abstract
Unsupervised semantic segmentation aims to assign category labels to every pixel in an image without the need for human-annotated ground truth. While self-supervised representation learning has seen significant success at the image level, pixel-level representations remain a challenging frontier. This paper presents a novel framework that learns pixel embeddings by grounding them in "visual concepts"—interpretable segments representing parts, objects, or scenes. By modeling the relationships between pixels and local concepts, local and global concepts, and the co-occurrence of these concepts, the proposed method achieves state-of-the-art performance on benchmarks like PASCAL VOC 2012 and COCO 2017.
Introduction
Semantic segmentation is a fundamental task in computer vision, yet its reliance on dense, pixel-wise annotations makes it prohibitively expensive to scale. Self-supervised learning (SSL) offers a potential solution by leveraging unlabeled data. However, existing unsupervised segmentation methods often struggle to bridge the gap between local appearance and global semantics.
Most current approaches rely on local affinity, ensuring that neighboring pixels with similar features are grouped together. While effective for boundary detection, this strategy often fails to recognize that the same semantic object (e.g., a "car") should have a consistent representation across different images with varying backgrounds and lighting. This paper addresses this limitation by introducing a global "visual vocabulary" that grounds pixel-level learning in shared semantic concepts.
Methodology: Grounding Pixels in Concepts
The core of the proposed method is a multi-level objective function that guides a Siamese network to learn robust pixel embeddings. The framework utilizes three primary types of supervision derived from "pseudo-segments"—initial image partitions generated by weak priors like contour detectors.
1. Pixel to Local Concepts
The model first ensures consistency within individual images. Using a local segmentation loss, it employs contrastive learning to force pixels within the same pseudo-segment to have similar embeddings while pushing pixels from different segments or different images apart. This establishes a baseline for local semantic coherence.
2. Local to Global Concepts
To achieve global consistency, the authors introduce a Vector Quantization (VQ) layer. A dictionary of "global concept" vectors is maintained across the entire dataset. Each local segment from an image is mapped to its nearest neighbor in this global dictionary. By minimizing the distance between local segments and their assigned global concepts, the model learns to cluster semantically similar regions (e.g., all "human faces") into the same discrete bins, regardless of which image they appear in.
3. Concept Co-occurrence
The model further refines these embeddings by exploiting the natural context of images. Certain concepts, such as "face," "hair," and "body," frequently appear together. The co-occurrence loss attracts the representations of pixels and segments whose associated global concepts often appear in the same image context. This encourages the model to learn a more holistic, context-aware representation of the visual world.
Experimental Results
The model was implemented using a PSPNet architecture with a ResNet-50 backbone. Training was performed primarily on the PASCAL VOC 2012 dataset, with evaluations extending to COCO 2017 and DAVIS 2017 to test generalization.
Benchmarking Performance
The proposed method significantly outperforms previous unsupervised baselines:
- PASCAL VOC 2012: The method achieved a 62.56 mIoU (using k-means clustering), representing a substantial improvement over the previous state-of-the-art, MaskContrast (49.67 mIoU).
- COCO 2017: On this more complex dataset, the model reached 38.00 mIoU, demonstrating that the learned "visual vocabulary" generalizes well to uncurated scenes.
- DAVIS 2017: In video instance mask tracking, the model achieved 50.4 J(Mean), outperforming several unsupervised video-based methods.
Qualitative Analysis
Visualizations of the segmentation masks show that the grounding in global concepts leads to much cleaner object boundaries and more consistent labeling across images compared to methods relying solely on local cues. The learned global concepts are often human-interpretable, corresponding to specific object parts (like wheels or windows) or entire categories (like birds or boats).
Discussion and Conclusion
This work demonstrates that grounding pixel-level SSL in a hierarchy of visual concepts is a powerful strategy for unsupervised semantic segmentation. By moving beyond local affinity and building a global visual vocabulary, the model captures the semantic consistency required for high-quality segmentation.
However, challenges remain. The method's performance is partially tied to the quality of the initial pseudo-segments. Furthermore, while the model excels at broad categories, it can still struggle with fine-grained distinctions (e.g., "cow" vs. "horse") without explicit supervision. Future research may focus on refining the pseudo-segmentation process and exploring ways to automatically discover more granular semantic hierarchies.
Summary updated