Multiple Fusion Adaptation: A Strong Framework for Unsupervised Semantic Segmentation Adaptation

Kai Zhang, Yifan Sun, Rui Wang, Haichang Li, Xiaohui HuarXiv 2021 · arXiv

Introduction

Semantic segmentation—the process of assigning a class label to every pixel in an image—is a critical component of modern computer vision systems. However, deep learning models for this task are highly sensitive to "domain shift." A model trained on synthetic data, such as images from video games like Grand Theft Auto V, often fails when deployed on real-world streets due to differences in lighting, weather, and texture.

Manually labeling real-world data is prohibitively expensive. Unsupervised Domain Adaptation (UDA) addresses this by adapting models trained on labeled "source" data to unlabeled "target" data. While self-training—where a model generates its own "pseudo-labels" to retrain on—is a popular UDA technique, it often suffers from a feedback loop of errors. The Multiple Fusion Adaptation (MFA) framework addresses this challenge by integrating three distinct information fusion strategies to create a more robust and accurate supervision signal.

The MFA Framework

The MFA framework utilizes a co-learning setup to mitigate the noise inherent in self-training. It integrates three primary fusion strategies:

  1. Cross-Model Fusion: The system employs two independent networks (Net A and Net B). Rather than a single model learning from its own potentially biased predictions, Net A provides supervision for Net B, and vice versa. This mutual disagreement prevents the models from converging on the same incorrect patterns.
  2. Temporal Fusion: To stabilize predictions, MFA uses a "Mean Net" for each learner—an exponential moving average of the model's weights over time. This smoothed version of the model provides more consistent and less jittery labels than the active model.
  3. Online-Offline Pseudo Label Fusion: This is the framework's core innovation. It combines stable "offline" labels (generated periodically) with "online" labels (generated on-the-fly by the co-learner's Mean Net).
The MFA architecture illustrating the interaction between Net A, Net B, and their respective Mean Nets across various fusion strategies.

Key Insights: Balancing Stability and Detail

A central finding of this research is the complementarity between offline and online labels. Offline labels are generated every few epochs; they are stable and class-balanced but tend to be "lazy," ignoring "hard" pixels where the model lacks high confidence. In contrast, online labels are generated for every mini-batch. Because they originate from the most current version of the model, they capture finer details and difficult regions that the offline process might filter out.

Qualitative comparison showing how online pseudo-labels capture more detail (e.g., poles and distant objects) compared to the sparser offline labels.

By fusing these two sources, the model maintains the stability required for long-term training while gaining the sensitivity needed to resolve complex scene elements like thin poles or distant pedestrians.

Experimental Results

The researchers evaluated MFA on two standard "Synthetic-to-Real" benchmarks: GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes. MFA achieved state-of-the-art (SOTA) results on both:

  • GTA5-to-Cityscapes: Reached 58.2% mIoU (mean Intersection over Union), surpassing previous leading methods like ProDA (57.5%).
  • SYNTHIA-to-Cityscapes: Reached 62.5% mIoU.

Ablation studies confirmed the synergy of the components. While basic self-training improved the initial model by 3.8%, the full MFA suite resulted in a 10.1% total improvement, proving that the three fusion strategies are mutually reinforcing rather than redundant.

Conclusion and Future Work

MFA provides a highly efficient, single-stage refinement tool for UDA. While it requires higher GPU memory to run two models simultaneously, it significantly reduces the need for multiple rounds of retraining. Future research may explore the application of this multi-fusion approach to "Real-to-Real" domain shifts, such as adapting models between different geographic locations or weather conditions.

Summary updated

BibTeX
@misc{zhang2021multiple,
  author = {Kai Zhang and Yifan Sun and Rui Wang and Haichang Li and Xiaohui Hu},
  title = {{Multiple Fusion Adaptation: A Strong Framework for Unsupervised Semantic Segmentation Adaptation}},
  year = {2021},
  eprint = {2112.00295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.00295},
}
Figure