Weighting Pseudo-Labels via High-Activation Feature Index Similarity and Object Detection for Semi-Supervised Segmentation

Prantik Howlader, Hieu Le, Dimitris Samaras

Abstract

Semi-supervised semantic segmentation methods leverage unlabeled data by generating pseudo-labels. However, the success of these methods hinges on the reliability of these labels. Existing approaches typically select high-confidence pixels to avoid errors, but high confidence does not guarantee correctness, particularly in early training stages. This paper proposes a novel two-step approach to improve pseudo-label reliability. First, we unify predictions from a trained object detector and a semantic segmentation model to identify reliable pixels. Second, we assign per-pixel learning weights based on the structural similarity between a pixel's feature vector and a class prototype, measured via robust rank-statistics. Our method integrates easily into existing frameworks and demonstrates significant improvements on the Cityscapes and Pascal VOC datasets.

Introduction: The Challenge of Confirmation Bias

In semi-supervised learning, models are trained on a small set of labeled data and a much larger pool of unlabeled data. The standard "teacher-student" paradigm involves a teacher model generating pseudo-labels for unlabeled images to supervise the student.

The primary obstacle in this process is confirmation bias. If a teacher model makes an error—assigning a high-confidence but incorrect label to a pixel—the student model learns this error as ground truth, reinforcing the mistake. While current methods attempt to mitigate this by filtering for "high-confidence" pixels, deep networks are often overconfident and poorly calibrated early in training. A pixel might boast a 95% confidence score yet be entirely misclassified, leading the model into a cycle of reinforced errors.

Methodology

To address these challenges, we introduce two primary innovations: an ensemble approach for pixel identification and a rank-statistics-based weighting system.

1. Reliable Pseudo-label Pixel Identification

Rather than relying solely on the segmentation model, we incorporate a separate object detector (Faster R-CNN) trained on the limited labeled data. A pixel is only deemed "reliable" if both the segmentation model and the object detector agree on its classification. This ensemble leverages the different inductive biases of the two architectures: the detector evaluates holistic object crops, while the segmenter focuses on local patches and surrounding context. This "second opinion" effectively filters out segmentation errors that might otherwise pass a simple confidence threshold.

2. Pseudo-label Pixel Weighting (PPW) via Rank-Statistics

Even among reliable pixels, not all labels are equally certain. We assign a weight to each pixel based on its similarity to a class prototype—an average feature representation of a class maintained in a memory bank.

The overall training pipeline showing the ensemble of segmentation and detection models, followed by the per-pixel weighting mechanism.

Traditional similarity metrics like cosine similarity depend on raw feature magnitudes, which fluctuate wildly during training. Instead, we utilize rank-statistics. We identify the indices of the top-k most activated dimensions in the feature vector. Because the structure of a feature vector (which neurons fire) is more stable than its intensity (how much they fire), this metric provides a robust signal even when the model is poorly calibrated. If a pixel's highest activations align with the dimensions of the corresponding class prototype, it receives a higher learning weight.

Experimental Results

We evaluated our approach by integrating it into four state-of-the-art (SOTA) semi-supervised frameworks: UniMatch, AugSeg, AEL, and U2PL. Testing was conducted on major benchmarks including PASCAL VOC 2012 and Cityscapes.

Performance Gains

Our method showed consistent improvements across all frameworks, particularly in low-data regimes:

  • PASCAL VOC: Using only 1/16 of available labels, our method improved the UniMatch and AugSeg baselines by 2.0% mIoU.
  • Cityscapes: For ResNet-101 based models in the 1/16 partition, we achieved gains of up to 3.4% mIoU over the AEL baseline.

Qualitative Analysis

Qualitative results demonstrate that our weighting system successfully down-weights misclassified pixels that conventional thresholding methods would have accepted. For example, in cases where a bird's wing or a computer case is mislabeled, our per-pixel learning weight correctly identifies these as low-reliability regions.

Qualitative comparison between conventional thresholding and our proposed per-pixel learning weight. Note how our method (d) correctly down-weights erroneous regions in the dashed boxes compared to conventional methods (c).

Discussion and Limitations

While highly effective, the method has specific constraints. The object detector ensemble is primarily effective for "object-centric" categories (e.g., cars, dogs). For "stuff" categories like sky or vegetation, which lack well-defined bounding boxes, the system relies on labeled pixels for prototypes. Additionally, training an auxiliary object detector and maintaining a feature memory bank introduces some computational overhead.

In conclusion, by shifting focus from feature magnitudes to feature activation structures and incorporating holistic object detection, we provide a more robust framework for semi-supervised segmentation that significantly mitigates the risks of confirmation bias.

Summary updated

Abstract

Semi-supervised semantic segmentation methods leverage unlabeled data by pseudo-labeling them. Thus the success of these methods hinges on the reliablility of the pseudo-labels. Existing methods mostly choose high-confidence pixels in an effort to avoid erroneous pseudo-labels. However, high confidence does not guarantee correct pseudo-labels especially in the initial training iterations. In this paper, we propose a novel approach to reliably learn from pseudo-labels. First, we unify the predictions from a trained object detector and a semantic segmentation model to identify reliable pseudo-label pixels. Second, we assign different learning weights to pseudo-labeled pixels to avoid noisy training signals. To determine these weights, we first use the reliable pseudo-label pixels identified from the first step and labeled pixels to construct a prototype for each class. Then, the per-pixel weight is the structural similarity between the pixel and the prototype measured via rank-statistics similarity. This metric is robust to noise, making it better suited for comparing features from unlabeled images, particularly in the initial training phases where wrong pseudo labels are prone to occur. We show that our method can be easily integrated into four semi-supervised semantic segmentation frameworks, and improves them in both Cityscapes and Pascal VOC datasets.
BibTeX
@inproceedings{howlader2024weighting,
  author = {Prantik Howlader and Hieu Le and Dimitris Samaras},
  title = {{Weighting Pseudo-Labels via High-Activation Feature Index Similarity and Object Detection for Semi-Supervised Segmentation}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=UlkhvCtJDc},
}
Figure