When several expert clinicians outline the same tumor or organ on the same scan, their outlines disagree, and the disagreement matters. A few millimeters decide whether radiation reaches all of a tumor or spills onto the heart, and whether quantitative texture features computed inside the outline mean anything at all. Current machine learning treats this disagreement as a statistical object. Each rater becomes an abstract latent vector fitted to their final masks. These models can reproduce the spread of expert opinion, but they cannot say what in the image each rater was responding to. We propose to borrow a tool from naturalistic neuroimaging. There, researchers predict each small brain region's response to rich, cluttered images with a linear readout on top of fixed deep-network features, then ask which feature families and which image patches drive that region. We apply the same recipe with each rater in place of the brain region. A per-rater linear readout on frozen, spatially dense image features predicts that rater's boundary. Comparing readouts across interpretable feature spaces then tells us whether a rater's deviation from the group tracks local intensity edges, textural context, or broader anatomical structure.
The second ingredient comes from the same neuroimaging literature's central lesson: behavior measured under stripped-down stimuli does not transfer to real-world conditions, and a person's visual tuning shifts with their task. Every multi-rater benchmark we know of collects annotations on isolated crops or slices. We therefore ask whether a rater's "tuning", meaning their readout weights, stays stable when the same expert delineates the same structures in a realistic workstation setting with slice scrolling and intensity-window adjustment. We test this within the same raters, which improves on the between-cohort design that earlier task-shift studies were forced into. The core claim is concrete. A large share of inter-observer disagreement that is not explained by simple global over- or under-segmentation is predictable from interpretable dense image features. That rater-specific feature weighting is (a) stable enough to transfer to a new rater from a few examples, and (b) measurably altered by viewing context beyond what repeat-annotation noise explains.
What Exists and What Is Missing
The multi-rater segmentation literature supplies the problem and the strongest competitors. Pionono represents each rater as a Gaussian in an 8-dimensional latent space. The mean of that Gaussian encodes the rater's systematic bias and the variance encodes their inconsistency. It reports unweighted Cohen's κ of 0.758 on Gleason 2019 and 0.711 on a 25-rater breast cancer set. The Latent Doctor Model infers a rater's style from a "support set" of their other annotations using a set encoder. It reports a KL divergence of 0.461 to the expert panel on Gleason 2019, and a Spearman correlation of up to 0.728 between model entropy and pathologist entropy on ISUP grading. Variational inference for inter-observer variability samples alternative segmentations from a 6-dimensional latent space and reaches a continuous Dice of 0.5372 on QUBIQ brain growth. All three learn latent codes solely from end-point masks. Those codes predict well but carry no statement about image content. Our proposal keeps the per-rater structure and replaces the opaque latent with readout weights over named feature spaces. The goal is explanation and transfer, not a new record on Q-Dice.
The method comes from BrainSAIL. BrainSAIL fits linear per-voxel readouts from dense Vision Transformer patch features to 7-Tesla fMRI responses to natural images. It removes patch-level noise with a learning-free procedure: it averages features over random flips, crops, and shifts. It also shows that different backbones carry different content, with self-supervised vision models leaning toward texture and vision-language models toward semantic categories. We reuse its feature-denoising step, its linear readout, and its backbone-contrast logic. BrainSAIL evaluates localization against fixed, objective masks, so multi-rater ambiguity is exactly the case it leaves out. The general encoding-model pipeline is described in Quantitative Modeling of the Visual System.
The ecological-validity lens comes from Visual semantic tuning across the cortex shifts between tasks. That study found tuning differences in 42.0 ± 6.3% of cortical vertices between passive movie viewing and active virtual driving. Its between-subject design forced group-average projections onto individuals. This result motivates our context experiment but does not constrain its outcome. Clinical viewing context might shift rater behavior a lot or not at all, and either answer is informative for how multi-rater benchmarks should be built.
Two gaze-based works are the nearest attempts to ground disagreement in perception. Inter-observer eye-gaze consistency uses regions where several observers' gaze agrees to regularize a mitosis detector. Gaze2Segment uses gaze to seed interactive segmentation. Both use gaze as a saliency or prompt signal for sparse or interactive tasks, and neither models why boundaries differ. We record gaze only as a covariate in the context study.
We deliberately do not scan clinicians' brains. The neuroimaging direction supplies the analysis method and the ecological-validity lens; it does not supply brain data. Recording fMRI during annotation would add cost without testing the claim above.
Core Mechanism
Let $s_r(i)\in[0,1]$ be rater $r$'s label at pixel $i$, and let $\bar s(i)$ be the mean over raters. The target we model is the rater's deviation $d_r(i)=s_r(i)-\bar s(i)$, restricted to a band around the consensus boundary where deviations occur. Let $\phi_k(i)$ be the denoised dense feature vector at pixel $i$ from feature space $k$. The feature spaces are a hand-built low-level space (intensity, gradient magnitude, local contrast at several scales), a self-supervised vision backbone, a vision-language backbone, and a signed-distance-to-consensus term. Each rater gets a ridge-regularized readout
$$\hat d_r(i)=\sum_k w_{r,k}^\top \phi_k(i)+b_r ,$$
fitted on training images and evaluated on held-out images. The signed-distance term together with the bias $b_r$ captures "this rater always draws a few pixels wider." Everything the feature spaces add beyond that term is disagreement tied to image content.
For each space $k$ we compute a unique explained variance, following standard variance partitioning in encoding studies:
$$U_{r,k}=R^2_r(\text{all})-R^2_r(\text{all}\setminus k),$$
where $R^2_r$ is held-out explained variance of $d_r$ for rater $r$. The vector $(U_{r,k})_k$ is the rater's interpretable profile. Projecting $w_{r,k}$ back onto patches, as BrainSAIL does for voxels, gives per-image maps of which regions pull that rater's boundary outward or inward.
Research Program
The first phase uses only existing multi-rater data: QUBIQ (challenge paper; QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge; prostate, brain growth, kidney, pancreas, and lesion tasks), Gleason 2019, and the 25-rater breast cancer set used by Pionono. On these we fit rater readouts and establish three things.
The first is how much held-out rater deviation the feature spaces explain beyond the geometric offset. The second is whether rater profiles cluster into recognizable styles, for example edge-followers versus context-extenders. We check whether these clusters agree with Pionono's latent clustering on the same raters, and whether profiles stay stable across the two backbone families. The third is transfer: we test whether a new rater's readout, fitted from $n$ support images, predicts their held-out masks as well as the Latent Doctor Model's support-set encoder given the same $n$.
The second phase is a new within-rater collection on one structure where disagreement is known to be clinically consequential. We choose prostate MRI because QUBIQ provides a matched public task. A small panel of radiologists annotates the same cases three times. The first pass uses isolated 2D slices with fixed windowing, matching benchmark conditions. The second pass uses a full-volume viewer with free scrolling, windowing, and access to the other MRI sequences, with eye tracking and viewer logs. The third pass repeats the isolated-slice condition after a washout period, which gives an intra-rater test–retest noise ceiling. Case order is counterbalanced across raters. The context question is whether the change in a rater's profile between the first and second passes exceeds the change between the first and third passes.
Baselines
The naive reference is a rater-agnostic model that predicts the consensus $\bar s$ for every rater, which gives $d_r=0$. By construction it scores zero explained deviation. It anchors what "rater-specific" means and gives the per-rater Dice floor. It uses the identical backbone and training data.
The most dangerous simple baseline is the per-rater geometric offset. Here each rater's mask is modeled as the consensus dilated or eroded by a fitted per-rater distance, optionally varying with local boundary curvature. This is the signed-distance term alone in our readout, fitted on the same training images with the same cross-validation. If image features do not add significant held-out $R^2$ over this baseline for most raters, pooled across QUBIQ tasks with a rater-level permutation test, our central claim is false.
The strongest existing methods are Pionono and the Latent Doctor Model, run on the same splits. For Pionono we use the reported κ of 0.758 on Gleason 2019 and 0.711 on the breast cancer set from the Pionono paper. For the Latent Doctor Model we use the reported KL of 0.461 on Gleason 2019 from the LDM paper. On QUBIQ brain growth, the variational latent model's 0.5372 continuous Dice is the reference from the VI paper. Calibration is judged against the ECE of 0.20 reported by Dice Semimetric Losses for weighted averaging with their loss on QUBIQ prostate. We will retrain these baselines ourselves with matched tuning budgets and not rely only on quoted numbers. To separate feature quality from head design, we also give Pionono's rater heads the same frozen backbone.
Beating these baselines does not mean higher κ. It means matching them in per-rater prediction, within the spread across seeds, while adding held-out explained deviation that the latent codes cannot attribute, and outperforming the support-set encoder in few-shot transfer at small $n$.
Three ablations isolate the key ingredients. The first shuffles rater identities across images; a stable profile should collapse under shuffling. The second drops single feature spaces, which is how $U_{r,k}$ is estimated. The third removes the BrainSAIL-style denoising, to check whether patch noise drives the attributions.
Why not just…?
Why not just average the raters' masks or train on the majority vote? That is our naive reference. It is known to discard the spread of opinion and to miscalibrate: majority voting gives an ECE of 0.51 on QUBIQ prostate versus 0.20 for soft averaging with semimetric losses in Dice Semimetric Losses. More fundamentally, averaging answers "what is the consensus" and our question is "why does rater $r$ depart from it." Averaging has no rater dimension at all.
Why not just use per-rater heads on a trained segmentation network, as Pionono effectively does? This is the real threat for prediction accuracy, and we expect it to match or beat our frozen-feature readouts on κ. We do not claim otherwise. The claim is that a readout over named feature spaces yields a decomposition of each rater's deviation that a learned latent cannot provide, and that this decomposition transfers across backbones and to new raters. The Pionono-on-frozen-features ablation and the few-shot transfer comparison settle whether interpretability comes at an accuracy cost. If profiles are unstable across backbones, the interpretive claim fails even if prediction is fine.
Why not just model rater bias as a global dilation or erosion? It may well explain most of the disagreement. Pionono's finding that latent space isolates chronic under-segmenters hints that much style is exactly this. That is why the geometric offset is built into our readout as the term every feature space must beat. A negative result here would itself be useful: it would say that multi-rater latent models are largely learning a scalar margin.
Why not just use eye tracking? Gaze tells us where a rater looked, not which property of the fixated tissue moved their pen. BrainSAIL's evidence that distinct features within one image region drive distinct responses argues that location alone underdetermines the cause. In the context study we test this directly by adding a gaze-density feature space to the readout. If gaze carries most of the unique variance of the context shift, the dense-feature account is redundant for that question.
Why not just use MC dropout or ensembles? They estimate uncertainty but produce no rater-specific hypotheses. MC dropout also trails latent models on QUBIQ brain growth, at 0.5083 versus 0.5372 continuous Dice in the VI paper. They answer a different question and appear only as calibration references.
Decisive Checks
Three results decide the project. First, on held-out QUBIQ, Gleason 2019, and breast cancer images, the dense feature spaces must add significant explained rater deviation beyond the geometric offset for a majority of raters. If they do not, the project reports that inter-observer disagreement in these benchmarks is mostly a global margin and stops there. Second, rater profiles fitted with the self-supervised backbone and with the vision-language backbone must agree in rank order of $U_{r,k}$ at well above shuffled-rater levels. If they do not, the attribution reflects the model more than the rater. Third, in the within-rater study, the profile change from isolated slices to the full workstation must exceed the slice test–retest change, measured in readout-weight distance and in per-rater Dice when a slice-trained readout is applied to workstation annotations. A positive result would mean current multi-rater benchmarks systematically mischaracterize clinical disagreement, the same lesson reductionist stimuli taught neuroimaging. A null result would license the field to keep using crop-based benchmarks with more confidence than it has today.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers
- Quantitative Modeling of the Visual System
- Visual semantic tuning across the cortex shifts between tasks
- Supporting Mitosis Detection AI Training with Inter-Observer Eye-Gaze Consistencies
- Gaze2Segment: A Pilot Study for Integrating Eye-Tracking Technology into Medical Image Segmentation
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Dice Semimetric Losses: Optimizing the Dice Score with Soft Labels