Pseudo-Label Noise Suppression Techniques for Semi-Supervised Semantic Segmentation

Sebastian Scherer, Robin Schön, Rainer Lienhart

TL;DR

This paper tackles the challenge of noisy pseudo-labels in semi-supervised semantic segmentation by introducing a novel loss weighting scheme that dynamically adjusts the influence of pseudo-labels based on model confidence. Their approach significantly improves performance, even with very limited labeled data, and shows promise for other computer vision tasks.

What Problem Does This Paper Address?

Semantic segmentation, a crucial task in computer vision for understanding image content at a pixel level, typically requires vast amounts of meticulously annotated data. This annotation process is incredibly time-consuming and expensive. Semi-supervised learning (SSL) offers a solution by leveraging abundant unlabeled data alongside limited labeled data. A common SSL technique involves using a model trained on labeled data to generate "pseudo-labels" for unlabeled images, which are then used to train a new model. However, these pseudo-labels are inherently noisy and error-prone, as they are generated by an imperfect model. Training with these noisy labels can degrade performance, creating a significant hurdle for effective SSL. This paper aims to mitigate the negative impact of these noisy pseudo-labels to unlock the full potential of SSL for semantic segmentation.

Methods

The authors propose a three-pronged approach to manage noisy pseudo-labels:

  1. Image Mixing Augmentation (CowMask): They employ a novel image mixing technique using a "CowMask" (inspired by CutMix) to create augmented training samples. This involves replacing parts of an unlabeled image with corresponding parts from another image, effectively creating occlusions and simulating an additional segmentation task. This helps to make the model more robust to errors in pseudo-labels.
  2. Dynamic Pseudo-Label Weighting: This is the core contribution. Instead of treating all pseudo-labels equally, they introduce a loss weighting scheme. The weight assigned to each pseudo-label is dynamically determined by the confidence of the student model (the model being trained) on that specific pseudo-label. This allows the model to learn from high-confidence pseudo-labels more strongly while down-weighting potentially incorrect low-confidence ones. They adapt the Symmetric Cross Entropy (SCE) loss to incorporate these dynamic weights.
  3. Pseudo-Label Filtering Analysis: They investigate the common practice of filtering out pseudo-labels with low confidence. While they analyze its effectiveness, their findings suggest that simple confidence-based filtering isn't sufficient, as correct pseudo-labels can sometimes have low confidence.

Datasets

The proposed methods are evaluated on three standard semantic segmentation datasets:

  • Cityscapes: A widely used dataset for urban scene understanding.
  • PASCAL VOC 2012: A classic dataset for object recognition and segmentation.
  • Mapillary: A large-scale dataset with diverse street-level imagery.

They also demonstrate the transferability of their findings to human pose estimation using the LSP dataset.

Key Insights and Ideas

  • Pseudo-labels are inherently noisy, and this noise significantly harms performance. The paper empirically confirms that removing noisy pseudo-labels can lead to substantial performance gains.
  • Simple confidence-based filtering of pseudo-labels is not a silver bullet. The authors found that correct pseudo-labels can sometimes have low confidence, meaning filtering them out can be detrimental.
  • Dynamic weighting based on the student model's confidence is more effective than static filtering. By allowing the model to learn how much to trust its own predictions (and thus the pseudo-labels), they can better manage noise.
  • Image mixing augmentation (CowMask) is a valuable addition, especially when combined with pseudo-labels. It improves robustness and performance, but doesn't help when ground truth is available.
  • The proposed methods are effective even with extremely limited labeled data. This is a critical finding for real-world applications where labeling is scarce.

Key Results

  • The proposed method achieves state-of-the-art performance on the Cityscapes, PASCAL VOC 2012, and Mapillary datasets.
  • On Cityscapes with only 15 labeled images, their approach significantly outperforms the supervised baseline and naive self-training. For instance, they achieve 66.5% mIoU with 15 labeled images, compared to a supervised baseline of 53.0%.
  • The dynamic weighting scheme improves performance across all experimental settings, with correct pseudo-labels receiving, on average, twice the weight of incorrect ones.
  • The methods also show positive results on the human pose estimation task, indicating their generalizability.

Limitations and Future Work

  • Confidence-based filtering has limitations: The paper highlights that confidence alone is not a perfect indicator of label correctness, and removing low-confidence labels can sometimes remove correct ones.
  • Further investigation into uncertainty estimation: While they explored confidence, more sophisticated uncertainty estimation methods could potentially yield better results for filtering.
  • Generalizability to other domains: While they showed promise in human pose estimation, further exploration across a wider range of computer vision tasks would be beneficial.
  • Hyperparameter sensitivity: The paper mentions sensitivity analysis for alpha and beta parameters of the SCE loss, suggesting that optimal performance might depend on careful tuning.

Summary updated

Abstract

Semi-supervised learning (SSL) can reduce the need for large labelled datasets by incorporating unlabelled data into the training. This is particularly interesting for semantic segmentation, where labelling data is very costly and time-consuming. Current SSL approaches use an initially supervised trained model to generate predictions for unlabelled images, called pseudo-labels, which are subsequently used for training a new model from scratch. Since the predictions usually do not come from an error-free neural network, they are naturally full of errors. However, training with partially incorrect labels often reduce the final model performance. Thus, it is crucial to manage errors/noise of pseudo-labels wisely. In this work, we use three mechanisms to control pseudo-label noise and errors: (1) We construct a solid base framework by mixing images with cow-patterns on unlabelled images to reduce the negative impact of wrong pseudo-labels. Nevertheless, wrong pseudo-labels still have a negative impact on the performance. Therefore, (2) we propose a simple and effective loss weighting scheme for pseudo-labels defined by the feedback of the model trained on these pseudo-labels. This allows us to soft-weight the pseudo-label training examples based on their determined confidence score during training. (3) We also study the common practice to ignore pseudo-labels with low confidence and empirically analyse the influence and effect of pseudo-labels with different confidence ranges on SSL and the contribution of pseudo-label filtering to the achievable performance gains. We show that our method performs superior to state of-the-art alternatives on various datasets. Furthermore, we show that our findings also transfer to other tasks such as human pose estimation. Our code is available at https://github.com/ChristmasFan/SSL_Denoising_Segmentation.
BibTeX
@misc{scherer2022pseudo,
  author = {Sebastian Scherer and Robin Schön and Rainer Lienhart},
  title = {{Pseudo-Label Noise Suppression Techniques for Semi-Supervised Semantic Segmentation}},
  year = {2022},
  eprint = {2210.10426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.10426},
}
Figure