Multi-rater Prompting for Ambiguous Medical Image Segmentation

Jinhong Wang, Yi Cheng, Jintai Chen, Hongxia Xu, Danny Ziyi Chen, Jian Wu

TL;DR

PU-Net introduces a parameter-efficient framework that uses "rater-aware prompts" to handle the inherent ambiguity in medical image segmentation. By fine-tuning only 0.3% of a model's parameters, it successfully learns to model both individual expert nuances and collective consensus, outperforming state-of-the-art methods while drastically reducing computational overhead.

What Problem Does This Paper Address?

In medical imaging, "ground truth" is often a moving target. When multiple experts (raters) annotate the same image—say, outlining a retinal disc—they rarely agree perfectly. Standard deep learning approaches typically collapse these diverse opinions into a single "majority vote" label. This process discards valuable information about diagnostic uncertainty and the specific expertise levels of different raters.

Furthermore, medical AI faces a "domain gap" problem: a model trained on data from one hospital often fails at another due to different equipment or lighting. Retraining massive models for every new site is computationally expensive and often impractical for clinical settings with limited hardware.

Methods

The authors propose PU-Net, a framework that treats different raters as distinct "input domains" using prompt learning.

  1. Rater-aware Prompts: Instead of retraining the whole network, the authors introduce learnable prompt tokens ($P_{r_1}, P_{r_2}, \dots, P_{r_R}$) for each individual rater, plus an additional aggregate prompt ($P_c$) for the consensus (majority vote).
  2. Implantable Transformer Blocks (ITBs): These blocks are inserted into a standard U-Net-like architecture with a CNN backbone. As shown in Figure 2, these blocks concatenate the rater-specific prompts with image features at various scales.
  3. Parameter-Efficient Fine-Tuning (PEFT): During training, the entire backbone is frozen. Only the prompts and the final segmentation head are updated. This accounts for only ~0.3% of the total parameters.
  4. Mix-training Strategy: The model is trained to predict individual rater labels when given a rater-specific prompt and the consensus label when given the aggregate prompt.
Fig. 2. An overview of our PU-Net architecture. The network is U-shaped and the backbone is a CNN module. We assign R + 1 prompts for R rater cases and apply prompt learning by inserting an Implantable Transformer Block (ITB) between every two adjacent downsampling/upsampling layers. Channel-wise FCs are used to align the dimension and distribution of the prompts with image features.

Datasets

The study primarily utilizes the RIGA dataset, a public benchmark for retinal cup and disc segmentation. It contains 750 color fundus images from three different sources (MESSIDOR, BinRushed, and Magrabia). Crucially, each image is manually labeled by six different glaucoma experts, providing the multi-rater data necessary to test the framework's ability to handle ambiguity.

Key Insights and Ideas

  • Disagreement as a Feature, Not Noise: By explicitly modeling each rater's "style" through prompts, the model learns the underlying distribution of plausible segmentations rather than just a single, potentially over-confident average.
  • Prompting for Uncertainty: Switching between different rater prompts allows the model to generate a range of predictions. The variance between these predictions serves as a natural proxy for clinical uncertainty.
  • Efficiency is Key: The ability to adapt a pre-trained model to a multi-rater scenario using only 0.1M trainable parameters (vs. ~30M for a full U-Net) makes this approach highly scalable for diverse clinical environments.

Key Results

PU-Net demonstrates superior performance on the RIGA dataset compared to both standard baselines and specialized multi-rater SOTA methods like CM-Net and MR-Net.

  • Quantitative Excellence: As shown in Table 1, PU-Net (prompt) achieved the highest Dice coefficients in almost every category. For the consensus (Majority-voting) case, it reached a Dice score of 96.58% for the disc and 85.70% for the cup, significantly higher than the baseline's 97.03/82.88.
  • Resource Efficiency: PU-Net achieved these results with only 0.10M trainable parameters, whereas the baseline required 29.94M and CM-Net required 14.3M.
  • Qualitative Calibration: Figure 3 illustrates that PU-Net's predictions for specific raters are much closer to the actual expert annotations than previous SOTA methods, which often struggle to capture individual rater nuances.
Fig. 3. Visualization of calibrated segmentation mask predictions of the SOTA method CM-Net and our proposed PU-Net. Raters 1–6 correspond to the annotations of 6 dif- ferent annotators, and MV corresponds to the majority-voting annotations. In the bottom row, the purple Prj or Pc in each box represents the prompt for the corre- sponding Rater j or MV.

Limitations and Future Work

While the paper demonstrates impressive results on 2D fundus images, the authors do not discuss the extension of this prompt-based approach to 3D medical data (like CT or MRI scans), which would involve significantly higher dimensional feature maps. Additionally, the current framework relies on having a fixed number of raters during training; adapting to a new, unseen rater at test time without any fine-tuning remains a challenge for future research.

Summary updated

Abstract

Multi-rater annotations commonly occur when medical images are independently annotated by multiple experts (raters). In this paper, we tackle two challenges arisen in multi-rater annotations for medical image segmentation (called ambiguous medical image segmentation): (1) How to train a deep learning model when a group of raters produces a set of diverse but plausible annotations, and (2) how to fine-tune the model efficiently when computation resources are not available for re-training the entire model on a different dataset domain. We propose a multi-rater prompt-based approach to address these two challenges altogether. Specifically, we introduce a series of rater-aware prompts that can be plugged into the U-Net model for uncertainty estimation to handle multi-annotation cases. During the prompt-based fine-tuning process, only 0.3% of learnable parameters are required to be updated comparing to training the entire model. Further, in order to integrate expert consensus and disagreement, we explore different multi-rater incorporation strategies and design a mix-training strategy for comprehensive insight learning. Extensive experiments verify the effectiveness of our new approach for ambiguous medical image segmentation on two public datasets while alleviating the heavy burden of model re-training.
BibTeX
@inproceedings{wang2024multi,
  author = {Jinhong Wang and Yi Cheng and Jintai Chen and Hongxia Xu and Danny Ziyi Chen and Jian Wu},
  title = {{Multi-rater Prompting for Ambiguous Medical Image Segmentation}},
  booktitle = {BIBM},
  year = {2024},
  url = {https://openreview.net/forum?id=EzCqCdvLan},
}
Figure