Abstract
Recent semi-supervised learning (SSL) methods are commonly based on pseudo labeling. Since the SSL performance is greatly influenced by the quality of pseudo labels, mutual learning has been proposed to effectively suppress the noises in the pseudo supervision. In this work, we propose robust mutual learning that improves the prior approach in two aspects. First, the vanilla mutual learners suffer from the coupling issue that models may converge to learn homogeneous knowledge. We resolve this issue by introducing mean teachers to generate mutual supervisions so that there is no direct interaction between the two students. We also show that strong data augmentations, model noises and heterogeneous network architectures are essential to alleviate the model coupling. Second, we notice that mutual learning fails to leverage the network's own ability for pseudo label refinement. Therefore, we introduce self-rectification that leverages the internal knowledge and explicitly rectifies the pseudo labels before the mutual teaching. Such self-rectification and mutual teaching collaboratively improve the pseudo label accuracy throughout the learning. The proposed robust mutual learning demonstrates state-of-the-art performance on semantic segmentation in low-data regime.
Introduction
Semi-supervised semantic segmentation aims to leverage a large pool of unlabeled data alongside a limited set of labeled examples. A prevalent strategy is pseudo-labeling, where a teacher model generates predictions for unlabeled images to supervise a student model. However, this approach is prone to "confirmation bias," where errors in the teacher's predictions are reinforced by the student. While mutual learning—where two models teach each other—was introduced to mitigate this, it often suffers from a "coupling issue." Direct interaction can cause models to converge into a homogeneous state, sharing the same errors and losing the diversity necessary for effective peer correction.
Methodology: Robust Mutual Learning
To address these challenges, we introduce Robust Mutual Learning (RML), which decouples the learning process and refines pseudo labels internally.
Indirect Mutual Learning (IML)
The core of RML is Indirect Mutual Learning. Instead of direct student-to-student interaction, we introduce Mean Teachers—models whose weights are an exponential moving average (EMA) of the students. Each student is supervised by the other's Mean Teacher, creating a temporal ensemble that buffers against immediate errors and maintains model diversity.
To further prevent model collapse, we incorporate:
- Data Noise: Aggressive augmentations like CutMix and RandAugment.
- Model Noise: Techniques such as Dropout and Stochastic Depth.
- Heterogeneous Architectures: Pairing different architectures (e.g., CNN and Transformer) to exploit varying inductive biases.
Pseudo Label Self-rectification
Beyond mutual teaching, we enable models to refine their own predictions. By calculating class-wise prototypes (centroids in the feature space), the model can re-evaluate its pseudo labels. If a pixel's features are closer to a different class prototype than its predicted label, the label is rectified or down-weighted, leveraging internal feature consistency to improve supervision quality.
Experimental Results
We evaluated RML on standard benchmarks, including Cityscapes and PASCAL VOC 2012. The framework demonstrates significant gains, particularly in extreme low-data scenarios.
On Cityscapes with only 1/30 labeled images (~100 images), RML achieved 58.25% mIoU, a 3.45% improvement over previous best methods. On PASCAL VOC 2012, using 1/8 of the data, the model reached 73.38% mIoU, nearing the performance of fully supervised models. Ablation studies confirm that both IML and self-rectification are critical, with IML alone providing a ~4% boost over baselines.
Conclusion
Robust Mutual Learning effectively tackles the coupling issue in semi-supervised segmentation through indirect teaching and internal label refinement. By maintaining model diversity and leveraging feature-space prototypes, RML sets a new state-of-the-art in low-data regimes, proving that diversity and self-correction are key to robust pseudo-labeling.
Summary updated