When four radiologists look at the same lung nodule, they disagree twice. They outline it differently, and they give it different malignancy scores on a five-point scale. Medical AI has studied these two disagreements separately. Segmentation work models the spread of outlines, and grading work models the spread of scores. Nobody has measured how much of the score disagreement is caused by the outline disagreement. This proposal measures it. We fine-tune a medical segmentation foundation model to reproduce each expert's own outline. Then we train a grading model that reads an image together with one expert's outline and predicts that same expert's score. Finally we swap outlines between experts while holding everything else fixed, and see how far the predicted grade moves. The result splits grade uncertainty into a part driven by the boundary and a part driven by interpretation.
The split matters for a concrete reason. Radiologists' malignancy scores on the LIDC-IDRI lung CT dataset are known to be pessimistic. A model trained on them reaches sensitivity 0.97 but specificity 0.23 against biopsy truth (Re-thinking and Re-labeling LIDC-IDRI). Another line of work argues that the "true" outline is whichever weighting of experts' outlines best supports diagnosis (DiFF). If over-staging comes mainly from generous outlines, then picking better outlines can fix it. If it comes mainly from interpretation, choosing outlines to fit clinical scores will only lock the pessimism in. Our decomposition, checked against biopsy-confirmed nodules, decides which of these is true.
Core Claim
Fine-grained staging is the setting where the coupling between boundary and grade should be strongest. Intermediate grades hinge on margin irregularity, size and spiculation, and those are exactly what experts outline differently. We claim three things.
- A large, measurable share of expert grade disagreement on LIDC-IDRI is carried by the expert's own outline.
- That share is concentrated at the intermediate scores (3 versus 4), not at the extremes.
- A grading model that conditions on outline distributions, rather than on a single consensus mask, is better calibrated to the expert panel. Once its boundary-driven component is controlled, it is also less biased against biopsy truth.
The decomposition uses the law of total variance. Let $x$ be the image, $m$ an expert outline drawn from the distribution of plausible outlines for $x$, and $y$ the malignancy score. Then
$$\mathrm{Var}(y \mid x) = \underbrace{\mathrm{Var}_{m}\big(\mathbb{E}[y \mid x, m]\big)}_{\text{boundary-driven}} + \underbrace{\mathbb{E}_{m}\big[\mathrm{Var}(y \mid x, m)\big]}_{\text{interpretation-driven}}.$$
We estimate the first term by holding the image fixed, passing each expert's outline through the grading model, and taking the variance of the predicted mean grades. The ratio of the first term to the total is the boundary share. We report it per case and per score level.
Why This Is Different From Existing Work
Several papers link two of the three directions, but none produces this measurement.
DiFF learns weights over multiple experts' masks that maximize a downstream diagnosis network's AUC. It uses CNNs trained from scratch and binary diagnoses: glaucoma, thyroid cancer, melanoma. It optimizes the outline for diagnosis. It never asks how much the diagnosis depends on the outline, and it never checks the diagnosis against pathology.
What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation? shows that experts agree less on malignant lesions, with mean Dice 0.753 versus 0.791 on benign ones. It uses predicted agreement as an auxiliary task to improve binary melanoma classification. Agreement there is a scalar summary. We instead pair each expert's outline with that same expert's grade.
The Latent Doctor Model models multi-pathologist grade distributions on Gleason and ISUP, but has no spatial component at all. D-Persona and PU-Net generate expert-specific outlines on LIDC-IDRI and RIGA. Both use U-Nets, and both evaluate only geometry.
On the foundation-model side, MedSAM-U builds uncertainty by jittering a single bounding box. U-MedSAM reweights the loss on single consensus masks. Neither touches real multi-expert data or grades.
What none of these does is use the within-case pairing that LIDC-IDRI provides: each reading supplies both an outline and a score from the same person. That pairing is what makes the counterfactual swap possible.
Research Program
The segmenter. We start from MedSAM. MedSAM is SAM retrained on about 1.57 million medical image–mask pairs, so much of the gap between natural photos and grayscale CT has already been closed. We add low-rank adapters to its image encoder. Low-rank adapters are small trainable matrices added beside frozen weights, in the spirit of Surgical-DINO. For slice-to-slice context we use 3D adapters as in MA-SAM.
We condition the segmenter on a per-reading latent code, so that sampling the code yields different plausible outlines. During training we use a loss that keeps samples between the intersection and the union of the expert masks, borrowed from D-Persona. LIDC-IDRI reader identities may not be consistent across scans, and we will verify this first. If they are not consistent, the code is a per-case latent rather than a persistent identity token. The decomposition only needs the within-case pairing, so it survives either way.
The grader. The grader is a small head on the same frozen-plus-adapter features. It pools features inside a given mask and outside it, adds simple shape descriptors of the mask, and outputs a distribution over the five scores. It is trained on triples of image, mask from reader k, and score from reader k.
At analysis time we compute the boundary-driven variance above in two ways. First with the actual expert masks, which is an exact counterfactual within the observed readings. Then with sampled masks, which shows whether the segmenter's spread suffices once human masks are absent.
The biopsy check. SCH-LND has 180 biopsy-confirmed solid nodules and no masks. For each nodule we generate a conservative outline (near the intersection of sampled outlines) and an aggressive one (near the union). We grade the nodule under each outline, and report sensitivity and specificity against pathology. If specificity rises substantially under conservative outlines, outline generosity is a real driver of the safety-first bias. If it barely moves, the bias is interpretive, and outline-calibration methods like DiFF cannot cure it.
Baselines
The naive reference is the standard pipeline on LIDC-IDRI. It uses a majority-vote consensus mask and the median score, with a classifier on image features. Its biopsy behaviour is already reported: sensitivity 0.97 and specificity 0.23 when trained on original labels (Re-thinking and Re-labeling LIDC-IDRI). Any method that does not move specificity on SCH-LND has not addressed the bias.
The strongest grade-side competitor is a label-distribution classifier. It is trained on the image alone with the empirical distribution of the experts' scores as a soft target, using the CVAE construction of the Latent Doctor Model. LDM reports KL divergence 0.461 to the expert panel on Gleason2019 and a Spearman correlation up to 0.728 with pathologist entropy on ISUP. No reported result was found on Lacuna for LDM on LIDC-IDRI malignancy scores, so we will run it ourselves.
The strongest biopsy-side competitor is the metric-learning relabeling of LIDC-IDRI against SCH-LND, reported at sensitivity 0.67 and specificity 0.66. We do not expect to beat relabeling outright, because it uses biopsy anchors directly. Instead we test whether our boundary-controlled grader closes a meaningful fraction of the gap between 0.23 and 0.66 without any biopsy labels. We also test whether combining it with relabeling helps.
The strongest segmentation competitor is D-Persona, which reports mean personalized Dice of 89.17% on LIDC-IDRI. No reported LIDC-IDRI generalized energy distance was found on Lacuna, so we will measure it. Generalized energy distance compares the whole set of predicted outlines with the whole set of expert outlines.
DiFF is re-run with five-level scores replacing its binary target, to test whether diagnosis-optimal mask weights still help at fine granularity.
All baselines share the same patient-level splits, the same 3D nodule crops, and an equal hyperparameter search budget of the same number of trials. Every method gets the same adapted MedSAM features wherever its architecture permits, with backbone-swapped variants reported separately.
The key ablation removes the mask input from our grader, so it sees only the image. If the mask-conditioned grader is no better calibrated to the panel than the image-only one, outline information adds nothing to prediction. The decomposition then reduces to a descriptive statistic.
Why not just…?
Why not just train an image-only classifier on soft expert labels? This is the real threat to the predictive claim. A model like LDM may match the score distribution without ever seeing an outline. If it does, the mask-ablation will show it, and we will report that outlines add no predictive calibration. The decomposition and the biopsy check still stand, because a soft-label classifier cannot say which part of its uncertainty comes from boundaries. It therefore cannot tell you whether to fix outlines or interpretations.
Why not just train nnU-Net or a 3D U-Net from scratch instead of adapting MedSAM? LIDC-IDRI nodules are small and plentiful enough that a specialist might match the foundation model. MedSAM's advantage was clearest on external data, with Dice gains of 15.5% to 52.3% on a nasopharynx task (MedSAM). The foundation model is not the core claim. We run the full pipeline with a 3D U-Net backbone and report both. The question that matters is whether the boundary share estimate is stable across backbones. If the estimate depends on the backbone, it is an artifact.
Why not just use MedSAM-U's box-jitter uncertainty as a stand-in for human disagreement? It needs no multi-expert data. Its value is that it rescues Dice from 63.3% to 83.8% under 50%-overlap boxes (MedSAM-U). Prompt sensitivity is not the same thing as expert disagreement, however. We test this directly. We correlate jitter-based mask entropy with the observed spread of expert masks and of expert scores, per nodule. We also substitute jitter samples for expert masks in the decomposition. If both correlations are high, cheap jitter suffices, and that is itself a useful finding.
Why not just apply DiFF to choose the best expert outline? DiFF optimizes outlines toward clinical labels. If those labels are pessimistic, it will select generous outlines. Our biopsy check measures whether DiFF-selected masks lower or raise specificity on SCH-LND, which directly tests the tension between DiFF and the LIDC relabeling result.
Proposed Experiments and Decisive Checks
Data. The primary data are LIDC-IDRI nodules with at least three readings, split by patient, plus SCH-LND for biopsy evaluation. IMA++ serves as an external check that our outline sampler tracks real disagreement. It has 15 annotators but only binary diagnosis, so it tests the segmenter, not the grade decomposition. AMPLIFAI offers granular LI-RADS staging and reports 10–30% boundary variability. It releases adjudicated rather than per-reader masks, so it cannot support the decomposition. We note it as the natural next target once per-reader liver annotations exist.
Metrics. For segmentation we use generalized energy distance and per-reader Dice. For grading we use KL divergence and expected calibration error against the panel score distribution. For the biopsy check we use sensitivity and specificity on SCH-LND. The decomposition outputs are the boundary share, with bootstrap intervals over patients, broken down by median score level.
Decisive check 1. The swap test on real expert masks gives a boundary share whose bootstrap interval excludes zero, and that share peaks at scores 3–4.
Decisive check 2. The mask-conditioned grader beats the image-only soft-label baseline on panel KL divergence.
Decisive check 3. Conservative outlines raise SCH-LND specificity above 0.23 without collapsing sensitivity.
If all three hold, outline disagreement is a lever on fine-grained staging bias. If the first holds but the third fails, over-staging lives in interpretation, and outline-calibration methods should be expected to fail on biopsy truth. That negative result would redirect the multi-expert segmentation line of work.
References
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- What Can We Learn from Inter-Annotator Variability in Skin Lesion Segmentation?
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Diversified and personalized multi-rater medical image segmentation
- Multi-rater Prompting for Ambiguous Medical Image Segmentation
- MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM
- U-MedSAM: Uncertainty-Aware MedSAM for Medical Image Segmentation
- Segment anything in medical images
- Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery
- MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation
- AMPLIFAI: A Multiphase CT Dataset for Benchmarking Clinical Reasoning in LI-RADS Assessment of Liver Lesions