Abstract
Training semantic segmentation models requires pixel-level annotations, leading to a significant labeling cost in dataset creation. To alleviate this issue, recent research has focused on semi-supervised learning, which utilizes only a small amount of annotation. In this context, pseudo labeling techniques are frequently employed to assign labels to unlabeled data based on the model’s predictions. However, there are fundamental limitations associated with the widespread application of pseudo labeling in this regard. Since pseudo labels are generally determined by the model’s predictions, these labels could be overconfidently assigned even for erroneous predictions, especially when the model has a confirmation bias. We observed that the overconfident prediction tendency of the cross-entropy loss exacerbates this issue, and to address it, we discover the focal loss, known for enabling more reliable confidence estimation, can complement the cross-entropy loss. The cross-entropy loss produces rich labels since it tends to be overconfident. On the other hand, the focal loss provides more conservative confidence, therefore, it produces a smaller number of pseudo labels compared to the cross-entropy. Based on such complementary mechanisms of two loss functions, we propose a simple yet effective pseudo labeling technique, Cross-Loss Pseudo Labeling (CLP), that alleviates the confirmation bias and lack of pseudo label problems. Intuitively, we can mitigate the overconfidence of the cross-entropy with the conservative predictions of the focal loss, while increasing the number of pseudo labels marked by the focal loss based on the cross-entropy. Additionally, CLP also contributes to improving the performance of the tail classes in class-imbalanced datasets through the class bias mitigation effect of the focal loss. In experimental results, our simple CLP improves mIoU by up to +10.4%p compared to a supervised model when only 1/32 true labels are available on PASCAL VOC 2012, and it surpassed the performance of the state-of-the-art methods.
Introduction
Semi-supervised learning (SSL) has emerged as a powerful paradigm for semantic segmentation, aiming to reduce the reliance on expensive pixel-level annotations. A common strategy in SSL is pseudo-labeling, where a model generates labels for unlabeled data to expand its training set. However, this approach is often plagued by confirmation bias. Models trained with standard Cross-Entropy (CE) loss tend to be overconfident, even when making incorrect predictions. These erroneous but high-confidence predictions become noisy pseudo-labels, leading the model to reinforce its own mistakes during training.
Methodology: Cross-Loss Pseudo Labeling (CLP)
To address the limitations of standard pseudo-labeling, we propose Cross-Loss Pseudo Labeling (CLP). The core innovation lies in leveraging the complementary strengths of two different loss functions: Cross-Entropy (CE) and Focal Loss (FL).
The Synergy of Opposites
We employ a single encoder with two distinct decoders:
- CE Decoder: Trained with standard Cross-Entropy loss. It generates a high volume of pseudo-labels ("rich" labels) but suffers from overconfidence and lower accuracy.
- FL Decoder: Trained with Focal Loss. FL is designed to focus on hard samples, which leads to more conservative and reliable confidence estimates. While it produces fewer pseudo-labels ("poor" quantity), they are significantly more accurate.
Cross-Supervision and Dual-Filtering
Instead of self-training, the decoders supervise each other. The FL decoder provides high-quality guidance to the CE decoder, while the CE decoder helps the FL decoder overcome its label scarcity. To further ensure label quality, a dual-filtering mechanism is implemented: a pixel is only assigned a pseudo-label if both decoders agree on the class and the prediction confidence exceeds a predefined threshold.
Experimental Results
We evaluated CLP on the PASCAL VOC 2012 and Cityscapes benchmarks across various label-scarce partitions.
Performance Gains
On PASCAL VOC 2012, CLP demonstrated substantial improvements. With only 1/32 of the available labels, CLP achieved a +10.4%p mIoU increase over a purely supervised baseline. It consistently outperformed state-of-the-art methods like USRN and U2PL, particularly in the most constrained settings.
Tail Class Mitigation
One of the significant advantages of using Focal Loss is its inherent ability to handle class imbalance. CLP significantly improved performance on "tail classes"—rare objects that are often overlooked by standard models. In our tests, CLP achieved a 2.5%p higher Tail-mIoU compared to previous leading methods.
Conclusion
Cross-Loss Pseudo Labeling (CLP) offers a simple yet effective solution to the confirmation bias problem in semi-supervised semantic segmentation. By pitting the "rich but noisy" predictions of Cross-Entropy against the "sparse but accurate" predictions of Focal Loss, CLP achieves a superior balance of label quantity and quality. While performance on high-resolution datasets like Cityscapes remains competitive with existing methods, the dramatic gains in label-scarce and class-imbalanced scenarios highlight CLP's potential for reducing annotation costs in real-world applications.
Summary updated