TL;DR
PU-Net introduces a parameter-efficient framework that uses "rater-aware prompts" to handle the inherent ambiguity in medical image segmentation. By fine-tuning only 0.3% of a model's parameters, it successfully learns to model both individual expert nuances and collective consensus, outperforming state-of-the-art methods while drastically reducing computational overhead.
What Problem Does This Paper Address?
In medical imaging, "ground truth" is often a moving target. When multiple experts (raters) annotate the same image—say, outlining a retinal disc—they rarely agree perfectly. Standard deep learning approaches typically collapse these diverse opinions into a single "majority vote" label. This process discards valuable information about diagnostic uncertainty and the specific expertise levels of different raters.
Furthermore, medical AI faces a "domain gap" problem: a model trained on data from one hospital often fails at another due to different equipment or lighting. Retraining massive models for every new site is computationally expensive and often impractical for clinical settings with limited hardware.
Methods
The authors propose PU-Net, a framework that treats different raters as distinct "input domains" using prompt learning.
- Rater-aware Prompts: Instead of retraining the whole network, the authors introduce learnable prompt tokens ($P_{r_1}, P_{r_2}, \dots, P_{r_R}$) for each individual rater, plus an additional aggregate prompt ($P_c$) for the consensus (majority vote).
- Implantable Transformer Blocks (ITBs): These blocks are inserted into a standard U-Net-like architecture with a CNN backbone. As shown in Figure 2, these blocks concatenate the rater-specific prompts with image features at various scales.
- Parameter-Efficient Fine-Tuning (PEFT): During training, the entire backbone is frozen. Only the prompts and the final segmentation head are updated. This accounts for only ~0.3% of the total parameters.
- Mix-training Strategy: The model is trained to predict individual rater labels when given a rater-specific prompt and the consensus label when given the aggregate prompt.
Datasets
The study primarily utilizes the RIGA dataset, a public benchmark for retinal cup and disc segmentation. It contains 750 color fundus images from three different sources (MESSIDOR, BinRushed, and Magrabia). Crucially, each image is manually labeled by six different glaucoma experts, providing the multi-rater data necessary to test the framework's ability to handle ambiguity.
Key Insights and Ideas
- Disagreement as a Feature, Not Noise: By explicitly modeling each rater's "style" through prompts, the model learns the underlying distribution of plausible segmentations rather than just a single, potentially over-confident average.
- Prompting for Uncertainty: Switching between different rater prompts allows the model to generate a range of predictions. The variance between these predictions serves as a natural proxy for clinical uncertainty.
- Efficiency is Key: The ability to adapt a pre-trained model to a multi-rater scenario using only 0.1M trainable parameters (vs. ~30M for a full U-Net) makes this approach highly scalable for diverse clinical environments.
Key Results
PU-Net demonstrates superior performance on the RIGA dataset compared to both standard baselines and specialized multi-rater SOTA methods like CM-Net and MR-Net.
- Quantitative Excellence: As shown in Table 1, PU-Net (prompt) achieved the highest Dice coefficients in almost every category. For the consensus (Majority-voting) case, it reached a Dice score of 96.58% for the disc and 85.70% for the cup, significantly higher than the baseline's 97.03/82.88.
- Resource Efficiency: PU-Net achieved these results with only 0.10M trainable parameters, whereas the baseline required 29.94M and CM-Net required 14.3M.
- Qualitative Calibration: Figure 3 illustrates that PU-Net's predictions for specific raters are much closer to the actual expert annotations than previous SOTA methods, which often struggle to capture individual rater nuances.
Limitations and Future Work
While the paper demonstrates impressive results on 2D fundus images, the authors do not discuss the extension of this prompt-based approach to 3D medical data (like CT or MRI scans), which would involve significantly higher dimensional feature maps. Additionally, the current framework relies on having a fixed number of raters during training; adapting to a new, unseen rater at test time without any fine-tuning remains a challenge for future research.
Summary updated