Introduction
Domain Adaptive (DA) semantic segmentation aims to bridge the gap between a labeled source domain (often synthetic data) and an unlabeled target domain (real-world imagery). A common strategy involves using a black-box model to generate "pseudo-labels" for the target domain to facilitate self-training. However, this process is frequently undermined by two types of noise: closed-set noise, where known classes are confused (e.g., a car labeled as a truck), and open-set noise, where objects unique to the target domain are forced into source-domain categories.
SimT (Simplex noise Transition matrix) addresses these challenges by explicitly modeling the noise distribution, allowing models to learn effectively from imperfect pseudo-labels rather than simply discarding low-confidence data.
The Challenge of Open-Set Noise
In real-world scenarios, the target domain often contains "unknown" classes—objects never encountered during source training. For example, a model trained on synthetic urban data may encounter specific types of trailers or ego-vehicles in a real-world dataset like Cityscapes that were not present in the original training set. Standard DA methods typically force these unknown objects into the most similar known category, creating unreliable training signals. While many researchers attempt to filter out these noisy pixels, this often removes valuable information near decision boundaries. SimT instead seeks to model this noise through a geometric lens, correcting the loss function to accommodate both known and unknown classes.
The SimT Framework
The core innovation of this research is the Simplex noise Transition matrix. Unlike traditional noise-modeling approaches that use a square $C \times C$ matrix (mapping $C$ classes to $C$ classes), SimT utilizes a non-square matrix. This matrix maps $C+n$ ground truth classes—where $n$ represents potential open-set classes—to the $C$ noisy pseudo-label categories provided by the black-box model.
To estimate this matrix without access to ground truth labels, the authors employ three geometric regularizers:
- Volume Regularization: Utilizing computational geometry, this minimizes the volume of the simplex formed by the matrix rows. This encourages the model to find the "tightest" possible fit for the underlying ground truth distribution.
- Anchor Guidance: The framework identifies "anchor points"—pixels that are highly representative of specific classes—to prevent the open-set portion of the matrix from collapsing into trivial or useless solutions.
- Convex Guarantee: Through a minimax optimization strategy, SimT ensures the transition matrix remains convex. This prevents the model from producing undiscriminating features where different classes become indistinguishable to the noise model.
Experimental Evaluation
The authors evaluated SimT across two distinct domains: urban driving and medical imaging.
Urban Driving (GTA5 → Cityscapes)
In the transition from synthetic GTA5 data to real-world Cityscapes images, the model must handle 19 shared classes and 15 open-set classes. SimT achieved state-of-the-art results in both Unsupervised Domain Adaptation (UDA) and Source-Free Domain Adaptation (SFDA) settings:
- UDA: Reached 58.9% mIoU, a 1.5% improvement over previous benchmarks.
- SFDA: Reached 54.4% mIoU, representing a significant 18.8% increase over the "Source Only" baseline.
Medical Imaging (Endovis17 → Endovis18)
In surgical instrument segmentation, the model adapted between different surgical datasets containing 3 shared and 3 open-set instrument types. SimT achieved 58.34% mIoU, significantly outperforming other domain adaptation methods like IGNet (53.74%).
Conclusion and Future Work
SimT demonstrates that treating label noise as a geometric problem allows for more robust domain adaptation. By embracing a non-square transition matrix, the model acknowledges the reality of unknown objects in target domains. Qualitative results show that SimT is particularly effective at identifying small-scale objects and differentiating between easily confused classes, such as "rider" versus "person."
While highly effective, the model currently requires a predefined hyperparameter $n$ for the number of open-set classes. Future research aims to introduce open-set class priors, which would allow the model to not only detect unknown regions but also semantically separate and identify individual open-set categories.
Summary updated