When several radiologists outline the same tumor, their contours disagree. Models that try to reproduce each expert's contour usually represent that expert as one fixed code, such as a learned vector or prompt, and inject it into the network the same way for every scan. That design assumes a rater's style is a constant offset, for example "Dr. A always draws 2 mm wide." Clinical experience suggests otherwise. The same expert may draw tight margins on large, well-contrasted tumors and generous ones on small infiltrative spots. If style depends on the case, a fixed code cannot represent it no matter how well it is trained. That would help explain the reported "expert ceiling," in which shared multi-rater models fall slightly short of a network trained on one rater alone.
This note proposes to measure first whether rater-by-case interaction actually exists in public multi-rater datasets, and how much of it can be predicted from the image. If it does, we build a segmentation network in which each rater's influence is applied through feature gates. These are per-channel multipliers between 0 and 1, computed fresh on every forward pass from both the rater's code and the current image. The idea fits in two sentences. Experts' habits change with the case, so the model that imitates an expert should let the expert's code interact with the image rather than add a constant. We test whether this interaction is real and predictable, and whether modeling it closes the gap to single-rater models at a fraction of their cost.
Core Claim
Disagreement among raters has three parts: a component shared across raters that depends on the case (genuine image ambiguity), a rater main effect (a constant style), and a rater-by-case interaction. Existing personalization methods model the first two well and the third only weakly or not at all. Our claim has two parts. The interaction carries a predictable share of rater deviation on real data. And a low-rank, input-conditioned gating of decoder channels captures that share better than a static rater code with the same parameter count.
The claim can be refuted. If image features do not predict per-rater deviation beyond the rater's mean offset on held-out cases, the interaction is intra-rater noise and the proposal should stop there. That would itself be a useful negative result for the field.
Relation to Prior Work
The prior work falls into three groups by how it injects rater identity.
The first group injects identity statically. Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability) gives each rater a Gaussian in an 8-dimensional latent space and concatenates a sample at the bottleneck. Its mean is a constant per-rater bias and its variance is intra-rater noise, so it has no slot for case-dependent style. ProSeg (Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization) cleanly separates image ambiguity from expert preference. However, it keeps the two latent variables as separate factors and does not let preference depend on the image.
The second group uses attention, which motivates our design and sets the bar we must beat. D-Persona (Diversified and Personalized Multi-Rater Medical Image Segmentation) uses per-expert cross-attention into a bank of latent codes and explicitly reports the expert ceiling. PU-Net (Multi-rater Prompting for Ambiguous Medical Image Segmentation) inserts transformer blocks in which learned rater prompts cross-attend with image tokens. Its prompts are static, but the attention gives some image-prompt interaction, so it is our most direct competitor. It also reports that it does not always match models trained on a single rater. MRNet (Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling) fuses an expertness vector with bottleneck features through a recurrent convolutional cell, assuming a fixed set of known raters. OMBA-UNet (Inter-Rater Uncertainty Quantification via Rater-Specific Bayesian Neural Networks) uses rater-specific attention gates but a full decoder per rater. Its parameters therefore grow linearly with the number of raters, and inference takes 0.34 s against 0.02 s for a plain U-Net.
We differ from all of these in two ways. We explicitly parameterize and test the interaction term. And each rater costs a short vector, not a decoder.
The third group supplies the method or constrains it. The gating mechanism follows dynamic channel gating in segmentation, as in the group-wise gating of GH-UNet, but conditions the gates on rater identity. The theory of single-layer attention in High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification shows that input-dependent weighting can detect weak, sparse signals that fixed linear pooling washes out. Rater-specific deviations on small lesions are exactly such a weak signal. The same fast adaptability is a warning, though: dynamic weights can lock onto idiosyncratic label patterns. That is why we keep the interaction low-rank and test it on held-out cases.
Beyond Boundary Noise shows that bottleneck features cluster by annotator disagreement even when output entropy does not. This suggests the image features we gate on already contain the information the interaction would need. Finally, the Latent Doctor Model (The Latent Doctor Model) infers a doctor's style from a support set of their annotated cases. That inspires our few-shot mode for unseen raters, but it works on classification and produces a global vector, not dense gating.
Research Program
Measurement study. On each dataset where rater identity is consistent across cases, we compute each rater's signed deviation from the leave-that-rater-out consensus for every case. The deviation is measured both as relative volume and as mean signed surface distance. We then compare two cross-validated predictors of the deviation on held-out cases. The first knows only the rater's training-set mean. The second also sees frozen image features from a consensus-trained network, using a small ridge regressor per rater.
The gain of the second predictor over the first is the predictable rater-by-case interaction. Without repeated readings, true interaction cannot be separated from intra-rater noise by variance decomposition alone. Out-of-sample predictability is what distinguishes structure from noise. We will also stratify the deviation by lesion size and contrast, to check the clinical intuition that a rater who is conservative on large lesions can be aggressive on small ones.
Model. The model is a single shared U-Net. At decoder scale $s$ it multiplies the feature map $F_s$ channel-wise by a gate
$$g_s(x, r) = \sigma\!\big(A_s e_r + U_s\,[(V_s e_r) \odot (W_s \bar f_s(x))]\big).$$
Here $e_r$ is a learned $d$-dimensional code for rater $r$, $\bar f_s(x)$ is the spatially pooled feature of image $x$ at that scale, $\sigma$ is the sigmoid, and $A_s, U_s, V_s, W_s$ are shared matrices. The product inside $U_s$ has rank $k$, which caps the capacity of the interaction. The first term is the rater's constant style. The second is the rater-by-case interaction. Deleting the second term yields a static rater-conditioned channel modulation, which serves as the exact ablation.
A spatial variant replaces $\bar f_s$ with per-location features so that gates can vary within a scan. We test it only if the channel version shows gains.
Each rater adds only $d$ parameters. The model therefore scales to cohorts like the 25-rater breast pathology set, unlike OMBA-UNet's per-rater decoders. For a new, unseen rater we freeze the network and fit $e_r$ by gradient descent on $n \in \{1, 3, 5, 10\}$ of that rater's annotated cases. For a deployment site with no rater annotations at all, we sample $e_r$ from the training raters' empirical distribution to produce a set of diverse hypotheses.
Baselines
The naive reference is a U-Net that ignores rater identity. It is trained on the pixel-wise mean of the raters' masks (soft labels), and its output is scored against every rater. Any personalization method must beat it on per-rater Dice, or it is not personalizing.
The upper reference, which defines the expert ceiling, is a separate U-Net of the same architecture trained only on each rater's masks. It uses the same epochs and tuning budget per model and therefore $m$ times the total compute. "Closing the ceiling" means getting within its confidence interval on per-rater Dice.
The strongest published competitor is D-Persona. On NPC-170 it reports a personalization mean Dice of 76.19% and a Generalized Energy Distance (a distance between predicted and real sets of masks) of 0.2359, against 0.4465 for the Probabilistic U-Net. On LIDC-IDRI it reports a personalization mean Dice of 89.17% (Diversified and Personalized Multi-Rater Medical Image Segmentation). ProSeg reports a Generalized Energy Distance of 0.1152 and a soft Dice of 91.53% on LIDC-IDRI, and a 20.71% diversity improvement on NPC (ProSeg).
PU-Net is the most direct mechanistic rival, since its prompts interact with the image through attention. On RIGA the file reports only its parameter efficiency, updating about 0.3% of parameters (roughly 0.1M), and gives no Dice. So there is no reported result on Lacuna for PU-Net segmentation accuracy on RIGA, and we will reimplement it. We will also run Pionono as the static-latent representative.
For fairness, all methods share the same backbone where the method allows, the same data splits, and the same augmentation. Each gets an equal hyperparameter search of 20 trials. Where possible, we match parameter counts to the total per-rater parameter budget of our model.
The central ablation compares the full gate against the same network with the interaction term removed, at identical $d$, with $k$ replaced by extra static capacity so parameter counts match. Beating D-Persona's reported NPC numbers matters less than one specific pattern. We need the interaction model to beat the static ablation on held-out cases. That gain should be concentrated in the strata where the measurement study found predictable interaction, especially small lesions.
Why not just…?
Why not just train one model per rater? It works, and it is our upper reference, but it costs $m$ full networks. It cannot use the anatomy shared across raters when each rater has few cases, and it says nothing about a new rater. Our claim is not that we beat it. The claim is that a shared model with a $d$-dimensional code per rater matches it, and that this fails without the interaction term. If the static ablation already matches it, the expert ceiling is a training-data artifact, not a missing interaction, and our idea is unnecessary. The ablation settles this directly.
Why not just use rater prompts with cross-attention, as PU-Net does, since attention already lets the prompt interact with the image? This is the real threat. Cross-attention between a static prompt and image tokens can in principle represent case-dependent style. We address it head on in three ways. We run PU-Net with its transformer blocks at matched parameters. We measure how much of the predictable interaction from the measurement study each model reproduces, using the correlation between predicted and true per-rater deviation on held-out cases. And we compare few-shot fitting of a new rater's code against PU-Net's prompt tuning at equal $n$. If PU-Net captures the interaction equally well, our contribution reduces to the measurement study and a cheaper, convolution-only mechanism, and we will report it that way.
Why not just give the static rater code more capacity, or inject it at every scale? A bigger static code still adds the same offset for every case. It cannot express "tight on large lesions, loose on small ones," because that style requires a product between rater and image features. The parameter-matched ablation tests exactly this. If the static version with extra capacity wins, the interaction hypothesis is wrong for these data.
Proposed Experiments
The measurement study comes first. It runs on NPC-170 (170 head-and-neck MRI scans, 4 radiologists) and RIGA (750 fundus images, 6 glaucoma specialists), where per-rater personalization is standard. We extend it to the QUBIQ tasks only where rater identity is consistent across cases. The study's output is the fraction of held-out per-rater deviation variance explained beyond the rater mean, with bootstrap intervals. We treat it as a go/no-go: if the explained fraction is indistinguishable from zero on both datasets, we stop the modeling work.
If the study passes, we train all baselines and our model on NPC-170, RIGA (training on MESSIDOR and BinRushed, testing on Magrabia, following PU-Net), and LIDC-IDRI. We report per-rater Dice, Generalized Energy Distance, and Q-Dice (Dice averaged over agreement thresholds 0.1 to 0.9, from QUBIQ). We also report the deviation-correlation metric stratified by lesion size, and the gap to the single-rater upper reference.
For new raters we use leave-one-rater-out on RIGA and NPC-170. Each held-out rater's code is fit from $n$ cases, and we plot per-rater Dice against $n$ for our model, PU-Net prompt tuning, and the static ablation. To guard against the fast-locking risk raised by the attention theory, we also report how the train-test gap behaves as rank $k$ grows. The intended operating point is the smallest $k$ at which held-out gains saturate.
Three outcomes together would confirm the proposal. The measurement study must show predictable interaction. The interaction model must beat its parameter-matched static ablation on held-out cases, mainly in the strata the measurement study flagged. And the remaining gap to the single-rater reference must shrink relative to D-Persona's and PU-Net's.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization
- Diversified and personalized multi-rater medical image segmentation
- Multi-rater Prompting for Ambiguous Medical Image Segmentation
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks
- GH-UNet: group-wise hybrid convolution-VIT for robust medical image segmentation
- High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification
- Beyond Boundary Noise: Aggregated Aleatoric Uncertainty Fails to Capture Presence Ambiguity in 3D Lung Nodule Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge