RWMS: Reliable Weighted Multi-Phase for Semi-supervised Segmentation

Wensi Liu, Xiao-Yu Tang, Chong Yang, Chunjie YangAAAI 2024 · OpenReview

Abstract

Semantic segmentation is a critical task in computer vision, yet the high cost of pixel-level annotations remains a significant barrier. Semi-supervised learning (SSL) offers a solution by leveraging both labeled and unlabeled data. In this work, we propose the Reliable Weighted Multi-Phase (RWMS) learning method. RWMS utilizes a self-training framework with two distinct teacher models to evaluate the reliability of pseudo-labels. By selecting reliable data at the image level and reweighting pseudo-labels at the pixel level, we guide multi-phase training toward more trustworthy knowledge. Combined with strong data augmentation, RWMS achieves remarkable performance, outperforming baseline methods by over 3% on the Pascal VOC and Cityscapes benchmarks.

The Challenge of Semi-Supervised Segmentation

Assigning a category to every pixel in an image is labor-intensive. While self-training—where a model generates "pseudo-labels" for unlabeled data to train itself—is a popular SSL strategy, it suffers from a fundamental flaw: noise. If a model generates an incorrect pseudo-label, it reinforces that error during training, leading to confirmation bias and degraded performance.

Existing solutions like co-training (CT) or dual-model training (DMT) attempt to mitigate this by using two models to cross-check each other. However, these often come with high computational costs or fail to fully exploit the consensus between different architectures. RWMS was designed to integrate the wisdom of dual models more efficiently while focusing the learning process on the most reliable data.

The RWMS Framework

The RWMS framework introduces three core innovations to stabilize the self-training process:

1. Dual-Teacher Reliability Assessment

RWMS employs two different teacher architectures (e.g., DeepLabv3+ and PSPNet). The core hypothesis is that simultaneous errors across different architectures are rare. Consensus is measured at two levels:

  • Image Level: The Mean Intersection-over-Union (MIoU) between the two teachers' predictions determines the overall "reliability" of an image.
  • Pixel Level: A reweighting formula, $w_i = ae^{-b(x-1)}$, is applied to each pixel. Here, the weight depends on how highly the second teacher ranks the first teacher's top prediction, ensuring the student model focuses on pixels where both teachers agree.

2. Multi-Phase Training

Instead of training on all unlabeled data simultaneously, RWMS uses a curriculum-based approach:

  • Stage 1: The model trains on labeled data and the top 50% most reliable unlabeled images (as determined by image-level MIoU).
  • Stage 2: Having established a solid foundation, the model generates higher-quality pseudo-labels for the remaining "unreliable" images and trains on the full dataset.

3. Strong Data Augmentation

To prevent the student from simply memorizing the teacher's output, aggressive augmentations—such as color jitter, blur, and grayscale transformations—are applied to unlabeled images. This forces the model to learn robust, invariant features rather than superficial patterns.

Comparison of RWMS with existing semi-supervised learning frameworks like DCT, CT, and DMT.

Experimental Results

The researchers evaluated RWMS on two standard benchmarks: Pascal VOC 2012 and Cityscapes.

  • Pascal VOC: With only 1/16th of the labels (~660 images), RWMS achieved 72.2% MIoU, a 3.4% improvement over the baseline. It consistently outperformed other state-of-the-art methods like DCC and ELN across various data splits.
  • Cityscapes: On the 1/30th labeling split, RWMS outperformed the DMT method by a substantial 6%, demonstrating strong generalization in complex urban environments.
  • Qualitative Success: Visual analysis indicates that RWMS produces significantly cleaner object boundaries and reduces "speckle" noise in complex regions compared to standard supervised or baseline semi-supervised approaches.

Conclusion and Future Work

RWMS effectively addresses the noise problem in self-training by using dual-teacher consensus as a proxy for truth. While the method requires an initial overhead to train the teacher models and involves hyperparameters for pixel weighting, its performance gains are significant. Future research may explore automated ways to determine the reliability split dynamically, further optimizing the curriculum as the student model evolves.

Summary updated

Abstract

Semantic segmentation is one of the tasks concerned in the field of computer vision. However, the cost of capturing large numbers of pixel-level annotations is expensive. Semi-supervised learning can utilize labeled and unlabeled data, providing new ideas for solving the problem of insufficient labeled data. In this work, we propose a data-reliability weighted multi-phase learning method for semi-supervised segmentation (RWMS). Under the framework of self-training, we train two different teacher models to evaluate the reliability of pseudo labels. By selecting reliable data at the image level and reweighting pseudo labels at the pixel level, multi-phase training is guided to focus on more reliable knowledge. Besides, we also inject strong data augmentations on unlabeled images while training. Through extensive experiments, we demonstrate that our method performs remarkably well compared to baseline methods and substantially outperforms them, more than 3% on VOC and Cityscapes.
BibTeX
@inproceedings{liu2024rwms,
  author = {Wensi Liu and Xiao-Yu Tang and Chong Yang and Chunjie Yang},
  title = {{RWMS: Reliable Weighted Multi-Phase for Semi-supervised Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=iRSwiFfbZq},
}
Figure