TAX: Tendency-and-Assignment Explainer for Semantic Segmentation with Multi-Annotators

Yuan-Chia Cheng, Zu-Yun Shiau, Fu-En Yang, Yu-Chiang Frank Wang

Introduction

Semantic segmentation models often operate as "black boxes," making it difficult to understand the reasoning behind pixel-level classifications. This opacity is particularly problematic in high-stakes domains like medical imaging, where multiple experts may provide conflicting labels based on individual "labeling tendencies"—such as a preference for over-segmenting or under-segmenting specific features. Standard models typically aggregate these labels into a single "ground truth," losing the nuance of individual expert styles.

The Tendency-and-Assignment Explainer (TAX) framework addresses this by identifying which annotator’s style the model is mimicking for specific image regions ("who") and providing visual prototypes to justify that choice ("why"), all without sacrificing predictive performance.

The TAX Framework

The TAX framework integrates two primary components into existing segmentation architectures like U-Net or DeepLabv3+:

  1. Annotator-Dependent Convolution Kernels: Instead of using uniform weights, the model learns unique subsets of convolution kernels for each annotator. These kernels capture individual stylistic tendencies, such as boundary smoothness or sensitivity to specific textures.
  2. Prototype-Bank Based Annotator Assigner ($G$): This component acts as the explanatory engine. It maintains a bank of trainable visual "prototypes"—representative image patches linked to specific annotators. During inference, the Assigner compares input image regions to these prototypes to generate an Annotator Mask, which dictates which annotator-specific kernels should be applied to which pixels.
The TAX framework architecture featuring the Annotator Assigner and Tendency Learning Layer.

Experimental Setup and Datasets

The researchers evaluated TAX across three datasets to test its robustness in both synthetic and real-world multi-annotator scenarios:

  • PASCAL VOC 2012: A standard benchmark where multi-annotator bias was simulated through morphological manipulations (dilation, erosion, and simplification).
  • EndoScene: A colonoscopy polyp dataset with synthetically generated annotator tendencies.
  • LIDC-IDRI: A real-world lung CT scan dataset containing labels from four different thoracic radiologists, providing a natural test for multi-annotator interpretability.

Results and Discussion

Experimental results demonstrate that TAX provides significant interpretability without compromising predictive performance. On the PASCAL VOC dataset, TAX achieved competitive mean Intersection-over-Union (mIoU) scores, performing on par with or better than non-interpretable baseline models. Furthermore, the Annotator Assigner reached 85.80% accuracy in correctly identifying the intended labeling tendency on the PASCAL VOC dataset.

Qualitative results showing the "Who" (Annotator Mask) and "Why" (Training Patch) explanations across different datasets.

Key insights from the study include:

  • Hybrid Tendencies: The model does not simply pick one annotator for the entire image; it can assign different annotator styles to different regions based on local image features.
  • Traceability: By utilizing a prototype bank, predictions can be traced back to specific training examples. This is vital for medical applications where understanding the "precedent" for a diagnosis is as important as the diagnosis itself.
  • Performance Stability: Adding explanatory layers does not degrade mIoU, proving that interpretability is not necessarily a "tax" on model accuracy.

Conclusion

TAX offers a robust solution for making semantic segmentation models more transparent. By explicitly modeling and explaining annotator tendencies, it moves beyond the "average" ground truth approach, providing a framework that respects and explains the inherent human diversity in training data. Future work may explore applying these techniques to deeper layers of the network or scaling the system for datasets with a very large number of annotators.

Summary updated

Abstract

To understand how deep neural networks perform classification predictions, recent research attention has been focusing on developing techniques to offer desirable explanations. However, most existing methods cannot be easily applied for semantic segmentation; moreover, they are not designed to offer interpretability under the multi-annotator setting. Instead of viewing ground-truth pixel-level labels annotated by a single annotator with consistent labeling tendency, we aim at providing interpretable semantic segmentation and answer two critical yet practical questions: "who" contributes to the resulting segmentation, and "why" such an assignment is determined. In this paper, we present a learning framework of Tendency-and-Assignment Explainer (TAX), designed to offer interpretability at the annotator and assignment levels. More specifically, we learn convolution kernel subsets for modeling labeling tendencies of each type of annotation, while a prototype bank is jointly observed to offer visual guidance for learning the above kernels. For evaluation, we consider both synthetic and real-world datasets with multi-annotators. We show that our TAX can be applied to state-of-the-art network architectures with comparable performances, while segmentation interpretability at both levels can be offered accordingly.
BibTeX
@misc{cheng2023tax,
  author = {Yuan-Chia Cheng and Zu-Yun Shiau and Fu-En Yang and Yu-Chiang Frank Wang},
  title = {{TAX: Tendency-and-Assignment Explainer for Semantic Segmentation with Multi-Annotators}},
  year = {2023},
  eprint = {2302.09561},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.09561},
}
Figure