Enhanced Soft Label for Semi-Supervised Semantic Segmentation

Jie Ma, Chuan Wang, Yang Liu, Liang Lin, Guanbin LiICCV 2023 · DOI

Abstract

Semi-supervised semantic segmentation (SSS) aims to reduce the heavy reliance on pixel-level annotations by leveraging unlabeled data. Traditional self-training methods often employ a hard confidence threshold to filter pseudo-labels, which inadvertently discards informative pixels in uncertain regions. This paper introduces the Enhanced Soft Label (ESL) framework, which addresses this "suboptimal trap" through two key innovations: Dynamic Soft Labels (DSL) and Pixel-to-Part Contrastive Learning. DSL preserves high-entropy supervisory signals by identifying dominant classes, while the contrastive module refines boundaries by learning from object parts. ESL achieves state-of-the-art performance on Pascal VOC 2012 and Cityscapes benchmarks.

1. Introduction

Semantic segmentation is a fundamental task in computer vision, yet its progress is hindered by the high cost of manual annotation. Semi-supervised learning (SSL) offers a solution by utilizing a small labeled set alongside a larger unlabeled pool. However, current "self-training" paradigms typically use a hard threshold (e.g., 0.95) to select pseudo-labels. This approach often ignores pixels where the model is uncertain, such as those on object boundaries.

Dominant classes in uncertain regions.

As shown in Figure 1, even when a model is not 100% confident in a single class, it can often narrow down the possibilities to a few "dominant classes." Discarding these pixels leads to confirmation bias and poor performance on complex boundaries.

2. Methodology

The ESL framework adopts a Teacher-Student architecture to maximize the utility of unlabeled data.

2.1 Dynamic Soft Labels (DSL)

Instead of a binary "keep or discard" decision, DSL identifies a subset of classes whose cumulative probability exceeds a threshold $\eta$. These classes are treated as a "soft" pseudo-label. This mechanism allows the model to learn from high-entropy pixels without being forced into potentially incorrect hard assignments, effectively balancing the learning process across both "easy" and "difficult" classes.

2.2 Pixel-to-Part Contrastive Learning

To counteract the potential blurriness of soft labels, ESL incorporates a pixel-to-part contrastive learning module. Unlike standard contrastive methods that pull pixels toward a coarse category prototype, ESL uses unsupervised K-means clustering to decompose categories into finer "object parts" (e.g., distinguishing a cat's head from its paws).

The ESL framework architecture.

As illustrated in Figure 3, the student model learns to pull pixel embeddings toward their corresponding part prototypes in a memory bank while pushing them away from others. This fine-grained supervision significantly sharpens the model's discriminative boundaries.

3. Experimental Results

The ESL framework was evaluated on the Pascal VOC 2012 and Cityscapes datasets under various labeling ratios.

  • Pascal VOC 2012: On the classic partition with only 92 labeled images (1/16), ESL achieved a mean Intersection over Union (mIoU) of 70.97%, outperforming the supervised baseline by over 25% and setting a new state-of-the-art.
  • Cityscapes: In urban scene segmentation, ESL maintained a consistent lead, reaching 75.12% mIoU at the 1/16 partition.

Qualitative analysis confirms that ESL produces cleaner, more accurate boundaries for complex objects like bicycles and pedestrians compared to previous methods.

4. Conclusion

The Enhanced Soft Label (ESL) framework demonstrates that uncertainty in pseudo-labeling is a valuable signal rather than mere noise. By combining Dynamic Soft Labels with fine-grained Pixel-to-Part Contrastive Learning, ESL effectively bridges the gap between supervised and semi-supervised performance. Future work will focus on optimizing the computational efficiency of the part-grouping process and exploring more advanced clustering techniques.

Summary updated

BibTeX
@inproceedings{ma2023enhanced,
  author = {Jie Ma and Chuan Wang and Yang Liu and Liang Lin and Guanbin Li},
  title = {{Enhanced Soft Label for Semi-Supervised Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00115},
}
Figure