Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions

David Bruggemann, Christos Sakaridis, Prune Truong, Luc Van Gool

TL;DR

Refign is a framework-agnostic extension for self-training-based Unsupervised Domain Adaptation (UDA) that leverages normal-condition "reference" images to improve semantic segmentation in adverse weather. By combining an uncertainty-aware alignment module with an adaptive label refinement mechanism, it sets new state-of-the-art benchmarks on datasets like ACDC and Dark Zurich without adding any training parameters or inference overhead.

What Problem Does This Paper Address?

Semantic segmentation—the task of assigning a class label to every pixel in an image—is a cornerstone of autonomous vehicle perception. While modern models perform exceptionally well in clear weather, they often fail catastrophically in adverse conditions like heavy rain, fog, or nighttime.

The primary bottleneck is data: pixel-level annotations for adverse conditions are incredibly scarce. Researchers typically turn to Unsupervised Domain Adaptation (UDA), training models on labeled clear-weather data (the "source") and adapting them to unlabeled adverse data (the "target"). However, many modern datasets also provide "reference" images—captures of the exact same scene as the target, but taken under favorable daytime conditions. The authors of Refign tackle the challenge of how to effectively use these reference images as a form of weak supervision to bridge the gap between clear and adverse domains.

Methods

Refign introduces a two-step process integrated into standard self-training UDA pipelines:

  1. Alignment (UAWarpC): Reference and target images are rarely perfectly aligned due to different viewpoints or moving objects. Refign uses a pre-trained, uncertainty-aware dense matching network called UAWarpC. This module warps the reference predictions to match the target's perspective and, crucially, generates a pixel-wise confidence map ($P_{\mathcal{R}}$). This allows the system to know which parts of the warped image are reliable and which (like dynamic cars or occlusions) should be ignored.
  2. Refinement: This step fuses the target's own predictions with the aligned reference predictions using an adaptive label correction mechanism.
  3. Trust Score ($s$): Based on the entropy of the target predictions, the model automatically relies more on the "easier" reference images early in training and becomes more independent as its performance on adverse data improves.
  4. Class-Specific Mixing: The authors observed that large static classes (like road or sky) are easier to align but often receive low confidence scores in textureless regions. They introduce a mask-based strategy to allow more aggressive mixing for these reliable classes.
The Refign training pipeline integrates alignment and refinement modules into a standard UDA setup.

Datasets

The authors evaluated Refign across several challenging benchmarks:

  • Cityscapes: Used as the primary source domain (clear weather).
  • ACDC: A large-scale dataset featuring fog, night, rain, and snow with corresponding reference images.
  • Dark Zurich: Focused on nighttime adaptation.
  • RobotCar & CMU Correspondence: Used for semi-supervised domain adaptation experiments.
  • MegaDepth: Used for pre-training the UAWarpC alignment module.

Key Insights and Ideas

  • Reference as a Bridge: Normal-condition images of the same scene act as an "intermediate domain." They share the sensor characteristics and geographic location of the target but the visual quality of the source, making them a perfect stepping stone for adaptation.
  • Uncertainty is a First-Class Citizen: By explicitly modeling warp uncertainty, Refign avoids the "confirmation bias" common in self-training. It can distinguish between a good alignment of a building and a bad alignment of a moving pedestrian.
  • Curriculum by Entropy: The use of a trust score based on prediction entropy (shown in Figure 5) naturally creates a learning curriculum. The model "listens" to the reference images more when it is most confused.
Trust scores decrease over time as the model becomes more confident in the target domain, with harder conditions (night) maintaining higher reliance on reference images for longer.

Key Results

Refign demonstrates significant performance boosts when added to existing UDA methods like DAFormer:

  • State-of-the-Art: On the ACDC test set, Refign-DAFormer achieved 65.5% mIoU, a massive +10.1% improvement over the baseline.
  • Nighttime Mastery: On Dark Zurich, it set a new record of 56.2% mIoU.
  • Efficiency: Because the alignment module is pre-trained and frozen, Refign adds zero new trainable parameters and introduces no overhead during inference, as the reference images are only needed during training.
  • Diversity: Compared to naive averaging of predictions, Refign's adaptive scheme preserves higher prediction diversity, which is critical for robust self-training.
Refign maintains higher prediction diversity compared to naive refinement schemes.

Limitations and Future Work

While highly effective, Refign still faces challenges with small static objects (like poles and traffic signs) and highly dynamic scenes, where alignment is inherently difficult. The authors suggest that future work could focus on even more nuanced handling of these categories. Additionally, the method currently relies on the availability of reference-target image pairs during the training phase, though such data is becoming increasingly standard in the field.

Summary updated

Abstract

Due to the scarcity of dense pixel-level semantic annotations for images recorded in adverse visual conditions, there has been a keen interest in unsupervised domain adaptation (UDA) for the semantic segmentation of such images. UDA adapts models trained on normal conditions to the target adverse-condition domains. Meanwhile, multiple datasets with driving scenes provide corresponding images of the same scenes across multiple conditions, which can serve as a form of weak supervision for domain adaptation. We propose Refign, a generic extension to self-training-based UDA methods which leverages these cross-domain correspondences. Refign consists of two steps: (1) aligning the normal-condition image to the corresponding adverse-condition image using an uncertainty-aware dense matching network, and (2) refining the adverse prediction with the normal prediction using an adaptive label correction mechanism. We design custom modules to streamline both steps and set the new state of the art for domain-adaptive semantic segmentation on several adverse-condition benchmarks, including ACDC and Dark Zurich. The approach introduces no extra training parameters, minimal computational overhead -- during training only -- and can be used as a drop-in extension to improve any given self-training-based UDA method. Code is available at https://github.com/brdav/refign.
BibTeX
@inproceedings{bruggemann2023refign,
  author = {David Bruggemann and Christos Sakaridis and Prune Truong and Luc Van Gool},
  title = {{Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00319},
}
Figure