Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation

Yuyang Zhao, Zhun Zhong, Na Zhao, Nicu Sebe, Gim Hee LeeZenodo 2023 · DOI · OpenAlex

Abstract

In this paper, we study the task of synthetic-to-real domain<br> generalized semantic segmentation, which aims to learn a model that<br> is robust to unseen real-world scenes using only synthetic data. The<br> large domain shift between synthetic and real-world data, including the<br> limited source environmental variations and the large distribution gap<br> between synthetic and real-world data, significantly hinders the model<br> performance on unseen real-world scenes. In this work, we propose the<br> Style-HAllucinated Dual consistEncy learning (SHADE) framework to<br> handle such domain shift. Specifically, SHADE is constructed based on<br> two consistency constraints, Style Consistency (SC) and Retrospection<br> Consistency (RC). SC enriches the source situations and encourages the<br> model to learn consistent representation across style-diversified samples.<br> RC leverages real-world knowledge to prevent the model from overfitting<br> to synthetic data and thus largely keeps the representation consistent<br> between the synthetic and real-world models. Furthermore, we present<br> a novel style hallucination module (SHM) to generate style-diversified<br> samples that are essential to consistency learning. SHM selects basis<br> styles from the source distribution, enabling the model to dynamically<br> generate diverse and realistic samples during training. Experiments show<br> that our SHADE yields significant improvement and outperforms stateof-<br> the-art methods by 5.05% and 8.35% on the average mIoU of three<br> real-world datasets on single- and multi-source settings, respectively.
BibTeX
@inproceedings{zhao2023style,
  author = {Yuyang Zhao and Zhun Zhong and Na Zhao and Nicu Sebe and Gim Hee Lee},
  title = {{Style-Hallucinated Dual Consistency Learning for Domain Generalized Semantic Segmentation}},
  booktitle = {Zenodo},
  year = {2023},
  doi = {10.5281/zenodo.7566098},
}

Loading related content...

Figure