Abstract
Semi-supervised semantic segmentation methods leverage unlabeled data by generating pseudo-labels. However, the success of these methods hinges on the reliability of these labels. Existing approaches typically select high-confidence pixels to avoid errors, but high confidence does not guarantee correctness, particularly in early training stages. This paper proposes a novel two-step approach to improve pseudo-label reliability. First, we unify predictions from a trained object detector and a semantic segmentation model to identify reliable pixels. Second, we assign per-pixel learning weights based on the structural similarity between a pixel's feature vector and a class prototype, measured via robust rank-statistics. Our method integrates easily into existing frameworks and demonstrates significant improvements on the Cityscapes and Pascal VOC datasets.
Introduction: The Challenge of Confirmation Bias
In semi-supervised learning, models are trained on a small set of labeled data and a much larger pool of unlabeled data. The standard "teacher-student" paradigm involves a teacher model generating pseudo-labels for unlabeled images to supervise the student.
The primary obstacle in this process is confirmation bias. If a teacher model makes an error—assigning a high-confidence but incorrect label to a pixel—the student model learns this error as ground truth, reinforcing the mistake. While current methods attempt to mitigate this by filtering for "high-confidence" pixels, deep networks are often overconfident and poorly calibrated early in training. A pixel might boast a 95% confidence score yet be entirely misclassified, leading the model into a cycle of reinforced errors.
Methodology
To address these challenges, we introduce two primary innovations: an ensemble approach for pixel identification and a rank-statistics-based weighting system.
1. Reliable Pseudo-label Pixel Identification
Rather than relying solely on the segmentation model, we incorporate a separate object detector (Faster R-CNN) trained on the limited labeled data. A pixel is only deemed "reliable" if both the segmentation model and the object detector agree on its classification. This ensemble leverages the different inductive biases of the two architectures: the detector evaluates holistic object crops, while the segmenter focuses on local patches and surrounding context. This "second opinion" effectively filters out segmentation errors that might otherwise pass a simple confidence threshold.
2. Pseudo-label Pixel Weighting (PPW) via Rank-Statistics
Even among reliable pixels, not all labels are equally certain. We assign a weight to each pixel based on its similarity to a class prototype—an average feature representation of a class maintained in a memory bank.
Traditional similarity metrics like cosine similarity depend on raw feature magnitudes, which fluctuate wildly during training. Instead, we utilize rank-statistics. We identify the indices of the top-k most activated dimensions in the feature vector. Because the structure of a feature vector (which neurons fire) is more stable than its intensity (how much they fire), this metric provides a robust signal even when the model is poorly calibrated. If a pixel's highest activations align with the dimensions of the corresponding class prototype, it receives a higher learning weight.
Experimental Results
We evaluated our approach by integrating it into four state-of-the-art (SOTA) semi-supervised frameworks: UniMatch, AugSeg, AEL, and U2PL. Testing was conducted on major benchmarks including PASCAL VOC 2012 and Cityscapes.
Performance Gains
Our method showed consistent improvements across all frameworks, particularly in low-data regimes:
- PASCAL VOC: Using only 1/16 of available labels, our method improved the UniMatch and AugSeg baselines by 2.0% mIoU.
- Cityscapes: For ResNet-101 based models in the 1/16 partition, we achieved gains of up to 3.4% mIoU over the AEL baseline.
Qualitative Analysis
Qualitative results demonstrate that our weighting system successfully down-weights misclassified pixels that conventional thresholding methods would have accepted. For example, in cases where a bird's wing or a computer case is mislabeled, our per-pixel learning weight correctly identifies these as low-reliability regions.
Discussion and Limitations
While highly effective, the method has specific constraints. The object detector ensemble is primarily effective for "object-centric" categories (e.g., cars, dogs). For "stuff" categories like sky or vegetation, which lack well-defined bounding boxes, the system relies on labeled pixels for prototypes. Additionally, training an auxiliary object detector and maintaining a feature memory bank introduces some computational overhead.
In conclusion, by shifting focus from feature magnitudes to feature activation structures and incorporating holistic object detection, we provide a more robust framework for semi-supervised segmentation that significantly mitigates the risks of confirmation bias.
Summary updated