SimT: Handling Open-set Noise for Domain Adaptive Semantic Segmentation

Xiaoqing Guo, Jie Liu, Tongliang Liu, Yixuan Yuan

Introduction

Domain Adaptive (DA) semantic segmentation aims to bridge the gap between a labeled source domain (often synthetic data) and an unlabeled target domain (real-world imagery). A common strategy involves using a black-box model to generate "pseudo-labels" for the target domain to facilitate self-training. However, this process is frequently undermined by two types of noise: closed-set noise, where known classes are confused (e.g., a car labeled as a truck), and open-set noise, where objects unique to the target domain are forced into source-domain categories.

SimT (Simplex noise Transition matrix) addresses these challenges by explicitly modeling the noise distribution, allowing models to learn effectively from imperfect pseudo-labels rather than simply discarding low-confidence data.

The Challenge of Open-Set Noise

In real-world scenarios, the target domain often contains "unknown" classes—objects never encountered during source training. For example, a model trained on synthetic urban data may encounter specific types of trailers or ego-vehicles in a real-world dataset like Cityscapes that were not present in the original training set. Standard DA methods typically force these unknown objects into the most similar known category, creating unreliable training signals. While many researchers attempt to filter out these noisy pixels, this often removes valuable information near decision boundaries. SimT instead seeks to model this noise through a geometric lens, correcting the loss function to accommodate both known and unknown classes.

The SimT Framework

The core innovation of this research is the Simplex noise Transition matrix. Unlike traditional noise-modeling approaches that use a square $C \times C$ matrix (mapping $C$ classes to $C$ classes), SimT utilizes a non-square matrix. This matrix maps $C+n$ ground truth classes—where $n$ represents potential open-set classes—to the $C$ noisy pseudo-label categories provided by the black-box model.

To estimate this matrix without access to ground truth labels, the authors employ three geometric regularizers:

  1. Volume Regularization: Utilizing computational geometry, this minimizes the volume of the simplex formed by the matrix rows. This encourages the model to find the "tightest" possible fit for the underlying ground truth distribution.
  2. Anchor Guidance: The framework identifies "anchor points"—pixels that are highly representative of specific classes—to prevent the open-set portion of the matrix from collapsing into trivial or useless solutions.
  3. Convex Guarantee: Through a minimax optimization strategy, SimT ensures the transition matrix remains convex. This prevents the model from producing undiscriminating features where different classes become indistinguishable to the noise model.

Experimental Evaluation

The authors evaluated SimT across two distinct domains: urban driving and medical imaging.

Urban Driving (GTA5 → Cityscapes)

In the transition from synthetic GTA5 data to real-world Cityscapes images, the model must handle 19 shared classes and 15 open-set classes. SimT achieved state-of-the-art results in both Unsupervised Domain Adaptation (UDA) and Source-Free Domain Adaptation (SFDA) settings:

  • UDA: Reached 58.9% mIoU, a 1.5% improvement over previous benchmarks.
  • SFDA: Reached 54.4% mIoU, representing a significant 18.8% increase over the "Source Only" baseline.

Medical Imaging (Endovis17 → Endovis18)

In surgical instrument segmentation, the model adapted between different surgical datasets containing 3 shared and 3 open-set instrument types. SimT achieved 58.34% mIoU, significantly outperforming other domain adaptation methods like IGNet (53.74%).

Conclusion and Future Work

SimT demonstrates that treating label noise as a geometric problem allows for more robust domain adaptation. By embracing a non-square transition matrix, the model acknowledges the reality of unknown objects in target domains. Qualitative results show that SimT is particularly effective at identifying small-scale objects and differentiating between easily confused classes, such as "rider" versus "person."

While highly effective, the model currently requires a predefined hyperparameter $n$ for the number of open-set classes. Future research aims to introduce open-set class priors, which would allow the model to not only detect unknown regions but also semantically separate and identify individual open-set categories.

Summary updated

Abstract

This paper studies a practical domain adaptive (DA) semantic segmentation problem where only pseudo-labeled target data is accessible through a black-box model. Due to the domain gap and label shift between two domains, pseudo-labeled target data contains mixed closed-set and open-set label noises. In this paper, we propose a simplex noise transition matrix (SimT) to model the mixed noise distributions in DA semantic segmentation and formulate the problem as estimation of SimT. By exploiting computational geometry analysis and properties of segmentation, we design three complementary regularizers, i.e. volume regularization, anchor guidance, convex guarantee, to approximate the true SimT. Specifically, volume regularization minimizes the volume of simplex formed by rows of the non-square SimT, which ensures outputs of segmentation model to fit into the ground truth label distribution. To compensate for the lack of open-set knowledge, anchor guidance and convex guarantee are devised to facilitate the modeling of open-set noise distribution and enhance the discriminative feature learning among closed-set and open-set classes. The estimated SimT is further utilized to correct noise issues in pseudo labels and promote the generalization ability of segmentation model on target domain data. Extensive experimental results demonstrate that the proposed SimT can be flexibly plugged into existing DA methods to boost the performance. The source code is available at https://github.com/CityU-AIM-Group/SimT.
BibTeX
@inproceedings{guo2022simt,
  author = {Xiaoqing Guo and Jie Liu and Tongliang Liu and Yixuan Yuan},
  title = {{SimT: Handling Open-set Noise for Domain Adaptive Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure