TL;DR
This paper tackles the challenge of adapting semantic segmentation models to new environments without labeled data. They introduce a novel "Pseudo-Label Refinement Network" (PRN) that actively corrects noisy pseudo-labels generated by a teacher model, significantly improving adaptation performance. This PRN, combined with a unique FFT-based training strategy and other enhancements, leads to state-of-the-art results on challenging domain adaptation benchmarks.
What Problem Does This Paper Address?
Deep learning models for semantic segmentation, which identify and classify every pixel in an image, typically require vast amounts of meticulously labeled data for training. However, when these models are deployed in real-world scenarios, the data they encounter often differs significantly from the training data (e.g., changes in lighting, weather, or camera sensors). This "domain shift" can drastically degrade performance. While Unsupervised Domain Adaptation (UDA) aims to bridge this gap using unlabeled target data, existing methods often struggle with "confirmation bias" – the propagation of errors from noisy pseudo-labels generated by the model itself, especially in the early stages of training. This paper addresses the critical need for more robust UDA methods that can effectively handle and mitigate the impact of these noisy pseudo-labels.
Methods
The core of this work is the Pseudo-Label Refinement Network (PRN). This auxiliary network is trained to perform two key tasks:
- Refine Pseudo-Labels: It corrects the predicted labels for target domain pixels.
- Localize Noisy Pixels: It predicts a binary "noise mask" indicating pixels likely to have incorrect pseudo-labels.
The PRN takes the segmentation logits (raw prediction scores) and features from a teacher model as input. A novel FFT-based perturbation strategy is employed for training the PRN. This involves:
- Perturbing Source Logits: By replacing the low-frequency amplitude components of source segmentation logits with those from a target image, the PRN is exposed to a form of "noisy" source data. This allows it to learn to refine labels even when the input is imperfect, using the ground-truth source labels as supervision.
- Perturbing Target Logits: A similar process is applied to target logits, enabling the PRN to learn to refine target pseudo-labels.
The PRN's noise mask is then used to selectively weight the cross-entropy loss for target data, effectively ignoring pixels predicted to be noisy.
In addition to the PRN, the authors incorporate two other components to further enhance adaptation:
- Contrastive Learning (CL): A pixel-wise contrastive loss is used to pull features of pixels belonging to the same class closer and push apart features of pixels from different classes. This is applied to both source (using GT labels) and target (using refined pseudo-labels) data.
- Fourier Adaptation (FA): This module synthesizes source images that mimic the style of target images, aiming to reduce the perceptual gap between domains without requiring additional training.
The overall framework trains a student network and the PRN simultaneously, with gradients from the PRN not flowing back to the student during its training, and vice-versa.
Datasets
The proposed method is evaluated on several standard UDA semantic segmentation benchmarks:
- Cityscapes → Dark Zurich: Adapting from daytime driving scenes (Cityscapes) to nighttime scenes (Dark Zurich).
- GTA → Cityscapes: Adapting from synthetic driving scenes (GTA) to real-world driving scenes (Cityscapes).
- SYNTHIA → Cityscapes: Adapting from another synthetic dataset (SYNTHIA) to real-world driving scenes (Cityscapes).
The paper also mentions using datasets like Cityscapes, Dark Zurich, GTA, and SYNTHIA for training and evaluation, with specific image counts and splits provided.
Key Insights and Ideas
- Directly addressing pseudo-label noise is crucial: The paper highlights that simply relying on confidence scores for pseudo-label selection is insufficient. An explicit mechanism to refine and identify noisy labels is more effective.
- FFT for robust training: Using Fourier Transforms to perturb image data (specifically, the amplitude of segmentation logits) is a clever way to create synthetic noise that mimics domain shifts, allowing the PRN to learn robust refinement strategies.
- PRN as a "denoiser" for self-training: The PRN acts as a learned filter, improving the quality of the pseudo-labels that guide the student model's learning, thereby mitigating confirmation bias.
- Complementarity of CL and FA: While the PRN is the main innovation, contrastive learning and Fourier-based style adaptation further boost performance, showing that these techniques can work synergistically with explicit noise refinement.
Key Results
The proposed method consistently outperforms state-of-the-art UDA semantic segmentation approaches across the evaluated benchmarks.
- GTA → Cityscapes: Achieved an mIoU of 71.3%, outperforming the previous best (DAFormer) by +3.1%. With HRDA training, it reached 75.0%.
- Cityscapes → Dark Zurich: Achieved an mIoU of 58.4%, significantly outperforming methods that use additional reference images and also outperforming the best prior method (Refign) by +2.2%.
- Ablation studies demonstrate the significant contribution of the PRN module, with the full model showing a +3.0% improvement over baseline self-training. The noise mask prediction also proved beneficial.
Qualitative results show that the PRN's predicted noise masks align well with visually noisy regions in teacher predictions, and the final segmentation maps generated by the proposed method are visually superior.
Limitations and Future Work
- Computational Overhead: The paper notes a ~24.7% increase in training time compared to the baseline DAFormer, although inference time remains the same. The FFT computation itself adds a minimal overhead (~4.9%).
- Future Work: The authors suggest integrating the masked image consistency module from MIC into their framework for further performance gains. They also mention that more qualitative examples and ablation studies are available in the supplementary material. The paper implicitly suggests that further exploration of different perturbation strategies and their impact on PRN training could be a fruitful direction.
Summary updated