TL;DR
Refign is a framework-agnostic extension for self-training-based Unsupervised Domain Adaptation (UDA) that leverages normal-condition "reference" images to improve semantic segmentation in adverse weather. By combining an uncertainty-aware alignment module with an adaptive label refinement mechanism, it sets new state-of-the-art benchmarks on datasets like ACDC and Dark Zurich without adding any training parameters or inference overhead.
What Problem Does This Paper Address?
Semantic segmentation—the task of assigning a class label to every pixel in an image—is a cornerstone of autonomous vehicle perception. While modern models perform exceptionally well in clear weather, they often fail catastrophically in adverse conditions like heavy rain, fog, or nighttime.
The primary bottleneck is data: pixel-level annotations for adverse conditions are incredibly scarce. Researchers typically turn to Unsupervised Domain Adaptation (UDA), training models on labeled clear-weather data (the "source") and adapting them to unlabeled adverse data (the "target"). However, many modern datasets also provide "reference" images—captures of the exact same scene as the target, but taken under favorable daytime conditions. The authors of Refign tackle the challenge of how to effectively use these reference images as a form of weak supervision to bridge the gap between clear and adverse domains.
Methods
Refign introduces a two-step process integrated into standard self-training UDA pipelines:
- Alignment (UAWarpC): Reference and target images are rarely perfectly aligned due to different viewpoints or moving objects. Refign uses a pre-trained, uncertainty-aware dense matching network called UAWarpC. This module warps the reference predictions to match the target's perspective and, crucially, generates a pixel-wise confidence map ($P_{\mathcal{R}}$). This allows the system to know which parts of the warped image are reliable and which (like dynamic cars or occlusions) should be ignored.
- Refinement: This step fuses the target's own predictions with the aligned reference predictions using an adaptive label correction mechanism.
- Trust Score ($s$): Based on the entropy of the target predictions, the model automatically relies more on the "easier" reference images early in training and becomes more independent as its performance on adverse data improves.
- Class-Specific Mixing: The authors observed that large static classes (like road or sky) are easier to align but often receive low confidence scores in textureless regions. They introduce a mask-based strategy to allow more aggressive mixing for these reliable classes.
Datasets
The authors evaluated Refign across several challenging benchmarks:
- Cityscapes: Used as the primary source domain (clear weather).
- ACDC: A large-scale dataset featuring fog, night, rain, and snow with corresponding reference images.
- Dark Zurich: Focused on nighttime adaptation.
- RobotCar & CMU Correspondence: Used for semi-supervised domain adaptation experiments.
- MegaDepth: Used for pre-training the UAWarpC alignment module.
Key Insights and Ideas
- Reference as a Bridge: Normal-condition images of the same scene act as an "intermediate domain." They share the sensor characteristics and geographic location of the target but the visual quality of the source, making them a perfect stepping stone for adaptation.
- Uncertainty is a First-Class Citizen: By explicitly modeling warp uncertainty, Refign avoids the "confirmation bias" common in self-training. It can distinguish between a good alignment of a building and a bad alignment of a moving pedestrian.
- Curriculum by Entropy: The use of a trust score based on prediction entropy (shown in Figure 5) naturally creates a learning curriculum. The model "listens" to the reference images more when it is most confused.
Key Results
Refign demonstrates significant performance boosts when added to existing UDA methods like DAFormer:
- State-of-the-Art: On the ACDC test set, Refign-DAFormer achieved 65.5% mIoU, a massive +10.1% improvement over the baseline.
- Nighttime Mastery: On Dark Zurich, it set a new record of 56.2% mIoU.
- Efficiency: Because the alignment module is pre-trained and frozen, Refign adds zero new trainable parameters and introduces no overhead during inference, as the reference images are only needed during training.
- Diversity: Compared to naive averaging of predictions, Refign's adaptive scheme preserves higher prediction diversity, which is critical for robust self-training.
Limitations and Future Work
While highly effective, Refign still faces challenges with small static objects (like poles and traffic signs) and highly dynamic scenes, where alignment is inherently difficult. The authors suggest that future work could focus on even more nuanced handling of these categories. Additionally, the method currently relies on the availability of reference-target image pairs during the training phase, though such data is becoming increasingly standard in the field.
Summary updated