Introduction
Semantic segmentation—the process of assigning a class label to every pixel in an image—is a critical component of modern computer vision systems. However, deep learning models for this task are highly sensitive to "domain shift." A model trained on synthetic data, such as images from video games like Grand Theft Auto V, often fails when deployed on real-world streets due to differences in lighting, weather, and texture.
Manually labeling real-world data is prohibitively expensive. Unsupervised Domain Adaptation (UDA) addresses this by adapting models trained on labeled "source" data to unlabeled "target" data. While self-training—where a model generates its own "pseudo-labels" to retrain on—is a popular UDA technique, it often suffers from a feedback loop of errors. The Multiple Fusion Adaptation (MFA) framework addresses this challenge by integrating three distinct information fusion strategies to create a more robust and accurate supervision signal.
The MFA Framework
The MFA framework utilizes a co-learning setup to mitigate the noise inherent in self-training. It integrates three primary fusion strategies:
- Cross-Model Fusion: The system employs two independent networks (Net A and Net B). Rather than a single model learning from its own potentially biased predictions, Net A provides supervision for Net B, and vice versa. This mutual disagreement prevents the models from converging on the same incorrect patterns.
- Temporal Fusion: To stabilize predictions, MFA uses a "Mean Net" for each learner—an exponential moving average of the model's weights over time. This smoothed version of the model provides more consistent and less jittery labels than the active model.
- Online-Offline Pseudo Label Fusion: This is the framework's core innovation. It combines stable "offline" labels (generated periodically) with "online" labels (generated on-the-fly by the co-learner's Mean Net).
Key Insights: Balancing Stability and Detail
A central finding of this research is the complementarity between offline and online labels. Offline labels are generated every few epochs; they are stable and class-balanced but tend to be "lazy," ignoring "hard" pixels where the model lacks high confidence. In contrast, online labels are generated for every mini-batch. Because they originate from the most current version of the model, they capture finer details and difficult regions that the offline process might filter out.
By fusing these two sources, the model maintains the stability required for long-term training while gaining the sensitivity needed to resolve complex scene elements like thin poles or distant pedestrians.
Experimental Results
The researchers evaluated MFA on two standard "Synthetic-to-Real" benchmarks: GTA5-to-Cityscapes and SYNTHIA-to-Cityscapes. MFA achieved state-of-the-art (SOTA) results on both:
- GTA5-to-Cityscapes: Reached 58.2% mIoU (mean Intersection over Union), surpassing previous leading methods like ProDA (57.5%).
- SYNTHIA-to-Cityscapes: Reached 62.5% mIoU.
Ablation studies confirmed the synergy of the components. While basic self-training improved the initial model by 3.8%, the full MFA suite resulted in a 10.1% total improvement, proving that the three fusion strategies are mutually reinforcing rather than redundant.
Conclusion and Future Work
MFA provides a highly efficient, single-stage refinement tool for UDA. While it requires higher GPU memory to run two models simultaneously, it significantly reduces the need for multiple rounds of retraining. Future research may explore the application of this multi-fusion approach to "Real-to-Real" domain shifts, such as adapting models between different geographic locations or weather conditions.
Summary updated