Abstract
Unsupervised semantic segmentation requires assigning a label to every pixel without any human annotations. Despite recent advances in self-supervised representation learning for individual images, unsupervised semantic segmentation with pixel-level representations is still a challenging task and remains underexplored. In this work, we propose a self-supervised pixel representation learning method for semantic segmentation by using visual concepts (i.e., groups of pixels with semantic meanings, such as parts, objects, and scenes) extracted from images. To guide self-supervised learning, we leverage three types of relationships between pixels and concepts, including the relationships between pixels and local concepts, local and global concepts, as well as the co-occurrence of concepts. We evaluate the learned pixel embeddings and visual concepts on three datasets, including PASCAL VOC 2012, COCO 2017, and DAVIS 2017. Our results show that the proposed method gains consistent and substantial improvements over recent unsupervised semantic segmentation approaches, and also demonstrate that visual concepts can reveal insights into image datasets.
Introduction
Semantic segmentation—the task of assigning a category label to every pixel in an image—is a cornerstone of computer vision. However, standard deep learning approaches are notoriously data-hungry, requiring thousands of hours of manual, pixel-perfect annotations. While self-supervised learning (SSL) has revolutionized image-level tasks, applying it to the pixel level remains difficult.
Existing pixel-level SSL methods often focus on local information within a single image, leading to a "semantic gap." For example, a model might recognize a patch of fur in two different images but fail to realize they both belong to the global concept of "dog" because it lacks a dataset-wide perspective. This paper introduces a framework that bridges this gap by grounding pixel representations in "visual concepts"—interpretable clusters of image segments that represent parts, objects, or scenes across an entire dataset.
Methodology: Grounding Visual Concepts
The core innovation of this work is a unified framework that uses three distinct relationships to guide the learning of pixel embeddings without human labels.
1. Pixel-to-Local Concepts
The model begins by generating "pseudo-segments" using traditional, non-learned algorithms such as HED contours or superpixels. These segments serve as initial, albeit noisy, semantic boundaries. The framework employs a contrastive loss to pull pixels within the same pseudo-segment together in the embedding space while pushing pixels from different segments apart. This ensures local spatial consistency.
2. Local-to-Global Concepts
To achieve consistency across the entire dataset, the authors utilize Vector Quantization (VQ). The model maintains a "dictionary" of global concept vectors. Each local segment extracted from an image is mapped to its nearest neighbor in this global dictionary. This process forces the model to learn representations that are consistent across different images—ensuring that a "wheel" in a truck photo is represented similarly to a "wheel" in a car photo.
3. Concept Co-occurrence
The authors observe that certain visual concepts naturally appear together, such as a human face, hair, and torso. They introduce a co-occurrence loss that encourages the model to associate these related concepts. This provides a higher-level semantic signal that helps the model group disparate parts into coherent objects without needing explicit category labels.
Experimental Setup
The architecture utilizes a standard PSPNet with a ResNet-50 backbone. The model was trained and evaluated on three major benchmarks:
- PASCAL VOC 2012: The primary benchmark for semantic segmentation.
- COCO 2017: Used to test the generalizability of the learned embeddings.
- DAVIS 2017: A video dataset used to evaluate instance mask tracking across frames.
Notably, the pixel embeddings were trained exclusively on PASCAL VOC and subsequently tested on COCO and DAVIS without further fine-tuning, demonstrating the robustness of the learned visual vocabulary.
Key Results and Insights
The proposed method establishes a new state-of-the-art for unsupervised semantic segmentation, significantly outperforming previous baselines:
- Benchmark Performance: On PASCAL VOC, the model achieved a 62.56% mIoU (mean Intersection over Union), a substantial leap over previous leaders like SegSort (49.17%) and MaskContrast (49.67%). On COCO, it achieved 38.00% mIoU.
- Video Tracking: In the DAVIS 2017 task of tracking object masks through video, the model achieved a 50.4 J (Mean) score, outperforming several recent video-specific unsupervised approaches.
- Interpretability: Unlike many "black box" SSL methods, the global concepts learned by this model are human-interpretable. Visualizations of the VQ dictionary show that the model naturally learns clusters corresponding to "faces," "animal legs," "wheels," and "water."
- The Power of Global Semantics: The results suggest that local pixel similarity is insufficient for high-quality segmentation. By introducing a global dictionary, the model avoids "over-segmenting" objects where parts look visually distinct (e.g., a car's shiny metal body vs. its black rubber tires).
Limitations and Future Work
While the method is highly effective, it remains somewhat dependent on the quality of the initial pseudo-segments. If the initial edge detection consistently fails to capture a specific type of boundary, the model may struggle to refine it. Additionally, without explicit labels, the model occasionally confuses visually similar semantic classes, such as "cows" and "horses," which share many visual concepts like legs, fur, and grassy backgrounds.
Future research may explore scaling this dictionary approach to even larger, more diverse datasets to determine if a "universal visual vocabulary" can be learned, further reducing the need for manual supervision in computer vision.
Summary updated