Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation

Yuhao Qing, Dan Zeng, Shaorong Xie, Kaer Huang, Yueying WangAAAI 2025 · DOI

TL;DR

This paper introduces IL2Vseg, a novel unsupervised semantic segmentation method that enhances existing approaches by integrating low-level visual cues (color, texture) with high-level features from self-supervised pre-trained models. This fusion leads to more accurate and continuous segmentation results, achieving state-of-the-art performance on several benchmark datasets.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a semantic label to every pixel in an image, is crucial for many computer vision applications. While supervised methods achieve high accuracy, they require extensive manual annotation, which is time-consuming and expensive. Unsupervised semantic segmentation aims to overcome this limitation by learning meaningful semantic groups without explicit labels. Existing unsupervised methods often rely on powerful self-supervised pre-trained models to extract high-level semantic features. However, these methods can overlook crucial low-level visual information like color and texture, leading to fragmented segmentations and inconsistencies, especially in complex scenes or for objects with subtle visual differences. This paper addresses the gap by proposing a method that effectively combines both high-level semantic understanding and low-level visual coherence for improved unsupervised segmentation.

Methods

The proposed IL2Vseg method has two main components:

  • High-Level Visual Cues: This component utilizes a pre-trained self-supervised backbone (e.g., DINO) to extract rich, high-level semantic features from the input images. These features are then projected into a lower-dimensional space.
  • Low-Level Visual Cues: To capture color and texture information that might be lost in high-level features, IL2Vseg introduces a novel approach based on spatially-constrained fuzzy clustering with color affinity.
  • Spatially Constrained Fuzzy Clustering: This part applies fuzzy C-means clustering while incorporating spatial information to ensure that neighboring pixels with similar characteristics are grouped together, thus improving local consistency and reducing noise sensitivity.
  • Color Affinity: A color affinity matrix is computed to quantify the similarity between pixel colors. This matrix is then used to boost the clustering results, ensuring that pixels with similar colors are more likely to belong to the same segment.

These low-level visual cues are then fused with the high-level features. To optimize this fusion and ensure semantic consistency, IL2Vseg employs:

  • Feature Similarity Loss: This loss function encourages the fused features to maintain both spatial structure and feature continuity, improving the clustering of similar regions.
  • Contrast Loss Functions: These losses, based on color and luminosity invariance, further enhance the learning of distinct features for different semantic categories and improve consistency across images.
  • Smoothing Loss and Correspondence Distillation Loss: These are also incorporated to promote smoother segment boundaries and improve accuracy.

Datasets

The paper evaluates IL2Vseg on the following datasets:

  • COCO-Stuff-27: A large-scale dataset derived from COCO, featuring 91 object categories and widely used for semantic segmentation.
  • Cityscapes: Focuses on urban environments, emphasizing vehicle and street object segmentation.
  • Potsdam-3: Contains high-resolution remote sensing images with urban annotations.
  • MaSTr1325: A maritime dataset for obstacle segmentation.

The authors mention merging categories from COCOStuff and Cityscapes to create 27 evaluation categories for their experiments.

Key Insights and Ideas

  • Low-level cues are critical for unsupervised segmentation: The paper strongly argues that while high-level features from self-supervised models are powerful, they are insufficient on their own. Integrating low-level cues like color and texture is essential for achieving coherent and accurate segmentations, especially for preserving fine details and spatial continuity.
  • Spatially-constrained fuzzy clustering with color affinity is effective: This novel approach for generating low-level cues successfully captures local image structure and color relationships, complementing the global semantic understanding from pre-trained models.
  • Fusion and optimization are key: Simply combining features is not enough. The proposed feature similarity loss and contrastive losses are crucial for effectively integrating and optimizing the fused features, leading to improved segmentation quality.
  • The weight of low-level cues matters: Ablation studies show that there's an optimal balance (weight $\alpha=0.5$) for integrating low-level cues; using only low-level cues or only high-level cues results in inferior performance.

Key Results

  • IL2Vseg achieves state-of-the-art performance on the COCO-Stuff and Cityscapes datasets, outperforming existing methods like TransFGU, STEGO, SmooSeg, and EAGLE in terms of accuracy (Acc) and mean intersection over union (mIoU).
  • On the COCO-Stuff dataset, IL2Vseg achieved an accuracy of 54.2% and mIoU of 20.1% (using ResNet50 backbone).
  • On the Cityscapes dataset, IL2Vseg achieved an accuracy of 83.0% and mIoU of 20.6% (using ViT-S/8 backbone).
  • The method also demonstrates superior performance on the MaSTr1325 dataset, with an accuracy of 93.6% and mIoU of 82.5%.
  • Qualitative results show that IL2Vseg produces more continuous and accurate semantic segments with better details compared to other methods, which often exhibit fragmented regions or incorrect category assignments.

Limitations and Future Work

  • Reliance on pre-trained models: The performance of IL2Vseg is still dependent on the quality of the features extracted by the self-supervised pre-trained backbone.
  • Computational cost: While not explicitly detailed as a limitation, the integration of fuzzy clustering and multiple loss functions likely adds to the computational complexity.
  • Future work: The paper suggests that further exploration of different low-level visual cues and their integration strategies could lead to even better performance. The authors also imply that their method can be extended to other unsupervised tasks.

Summary updated

BibTeX
@inproceedings{qing2025integrating,
  author = {Yuhao Qing and Dan Zeng and Shaorong Xie and Kaer Huang and Yueying Wang},
  title = {{Integrating Low-Level Visual Cues for Enhanced Unsupervised Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32708},
}
Figure