When four radiologists outline the same lung nodule on a CT scan, they draw four different boundaries. Current machine-learning models of this disagreement can reproduce it: they learn a hidden "style" vector per annotator and sample plausible masks from it. They cannot say why the masks differ. Separately, interpretable diagnostic models predict human-readable signs (spiculated margin, lobulation, ground-glass texture) before predicting malignancy. These models assume a single correct boundary and usually discard the cases where experts disagree. We propose to join the two. We build a segmentation model whose only route for annotator-to-annotator variation is a small set of clinical concepts. We then measure how much of real inter-observer boundary disagreement those concepts account for.
The opportunity is specific to the LIDC-IDRI lung nodule dataset. There, each radiologist supplies a mask together with their own ratings of nine semantic attributes and malignancy for the same nodule. That gives per-rater triples of boundary, concepts, and diagnosis. Existing multi-rater methods use only the masks, and existing concept models use only the averaged ratings. If a concept-conditioned generator can reproduce each radiologist's mask nearly as well as an unconstrained style vector, then disagreement has a clinical explanation. If it cannot, the size of the gap tells us how much disagreement lies outside the clinical vocabulary. Both outcomes are publishable and useful.
Core Claim
Inter-observer variability in nodule segmentation is largely explained by disagreement in how raters perceive a few named morphological attributes, chiefly margin, spiculation, lobulation, and texture. The test is a model that generates rater-specific masks conditioned on that rater's concept ratings plus a deliberately small unexplained residual. We claim this model matches the fit of an unconstrained latent-variable model to the distribution of real masks. It should also attribute most of the reconstruction gain to the concept channel rather than the residual. At test time, rater ratings are not available. The model therefore predicts a distribution over concept ratings from the image and samples masks through it, so every sampled boundary comes with a stated reason, such as "this contour assumes a poorly defined margin."
Relation to Prior Work
Four multi-rater segmentation methods motivate the proposal by leaving the "why" open.
-
Pionono models each annotator as an 8-dimensional Gaussian over a U-Net latent space. - The Latent Doctor Model embeds annotator style from a support set with a conditional variational autoencoder.
-
EDUE forces the spread among a network's multiple heads to match human disagreement maps.
- MRNet reconstructs individual raters' masks to build attention over ambiguous regions.
All four treat disagreement as statistical or spatial. We replace the unconstrained style variable with a concept variable and keep their probabilistic machinery. Pionono and MRNet also need rater identities or expertise vectors, which retrospective cohorts often lack. Our conditioning variable is the rater's own ratings, which travel with each annotation.
UGS-Net is the closest spatial precedent on the same data. It trains separately on the intersection and union of LIDC masks and shows that disagreement concentrates on low-density tissue and spiculated edges. It does not connect these zones to attributes or diagnosis. We test directly whether those zones are the ones predicted to move when the margin and spiculation concepts change.
On the interpretability side, PN-SAMP jointly predicts nodule masks, nine attributes, and malignancy. It reports 97.58% malignancy accuracy and 89.33% attribute accuracy, but only on the 1,404 of 2,660 nodules with clear consensus. We keep the discarded ambiguous cases because that is where explanation matters most. Lesion-centric latent phenotypes derive shape descriptors and pooled encoder features from a single mask, without asking how rater-dependent boundaries would change them. That question becomes one of our robustness tests.
The closest bridge is DiFF. It weights raters' masks by how much they help a diagnostic network, and its authors note that the resulting features are not tied to clinical biomarkers. We differ in mechanism: we explain variation through named concepts rather than reweighting it by an opaque classifier. We also differ in supervision target, because two papers act as constraints on DiFF's approach:
- Re-thinking and Re-labeling LIDC-IDRI shows that radiologist malignancy scores on LIDC carry a defensive bias. Models trained on them reach 0.97 sensitivity but 0.23 specificity against biopsy-proven SCH-LND nodules. We therefore treat malignancy scores as another rater concept rather than as truth, and we evaluate diagnosis only against pathology.
- DiFF's own spatial smoothing of expertness maps risks erasing high-frequency structure such as spiculation. For this reason, our concept channel acts on the full-resolution decoder.
Visual concept filtering shows that concept sets containing non-visual items hurt accuracy. On HAM-10000, pruning them raised 1-shot accuracy from 36.5% to 53.2%. We heed this result by restricting the concept channel to LIDC's visual attributes and excluding "subtlety" and "malignancy" from the conditioning variable.
Research Program
Model. For nodule image $x$, rater $r$ supplies a mask $y_r$ and an attribute vector $c_r$ (margin, spiculation, lobulation, sphericity, texture, internal structure, calcification). The model factorizes the rater's mask as
$$p(y_r \mid x) = \int\!\!\int p_\theta(y_r \mid x, c, z)\, p_\phi(c \mid x)\, p(z)\, dc\, dz .$$
The terms are as follows:
- $p_\theta$ is a 3D U-Net decoder with feature-wise modulation by $c$.
- $p_\phi(c \mid x)$ is an image-conditioned distribution over ordinal concept ratings, which captures how raters' perceptions spread.
- $z$ is a low-dimensional residual latent with a strong prior penalty. It absorbs variation the concepts cannot explain.
During training, $c$ is set to the observed $c_r$. The residual $z$ is inferred by a posterior network, as in a probabilistic U-Net or Pionono, but with a tight capacity budget on its KL term.
Measuring how much disagreement the concepts explain. We define it by comparing reconstruction quality across three conditioning settings. The first conditions on nothing rater-specific, giving the consensus fit. The second conditions on $c_r$ alone with $z$ zeroed. The third conditions on an unconstrained latent of equal total capacity, as in Pionono. The explained fraction is
$$E = \frac{D(c_r) - D(\varnothing)}{D(\text{free}) - D(\varnothing)},$$
where $D(\cdot)$ is the mean Dice between generated and true rater-specific masks under that conditioning. $E$ near 1 means concepts carry nearly all the rater-specific information an unconstrained latent can capture. We report $E$ overall and stratified by attribute agreement.
Counterfactual checks. We edit one concept, for example margin from "sharp" to "poorly defined," and record where the generated boundary moves. We then check whether these displacements fall inside UGS-Net-style union-minus-intersection zones. We also check whether the edits match real rater pairs that differ mainly in that attribute.
Diagnosis against pathology. A malignancy head reads only the sampled concepts and mask-pooled features, in the style of mask-weighted pooling. It is evaluated on biopsy-confirmed SCH-LND nodules, never against radiologist scores.
Baselines
The naive reference is a single 3D U-Net trained on majority-vote masks, paired with a PN-SAMP-style attribute head trained on mean ratings. It defines $D(\varnothing)$ and represents current practice. The file reports 73.89% Dice for PN-SAMP and 86.12% Dice for UGS-Net. These are on different nodule subsets and protocols (1,404 consensus nodules versus 1,859 multi-annotated nodules with 5-fold cross-validation), so we will retrain both under our split rather than compare against the reported numbers.
The strongest competitor for reproducing rater masks is a Pionono-style model with an unconstrained latent of the same total dimension as our concepts plus residual. It is the most direct statement that disagreement can be modeled without semantics. No reported result was found on Lacuna for Pionono on LIDC-IDRI; its reported figures are κ = 0.758 on Gleason 2019 and κ = 0.711 on the breast cancer histopathology dataset. Because LIDC does not link raters consistently across scans, we give this baseline a per-annotation posterior latent rather than per-rater identities. That puts it under the same metadata constraint as our model. Both models share backbone, augmentation, epochs, and a matched hyperparameter search of equal trials, on the same 5-fold nodule-level split over all multi-annotated nodules.
For diagnosis, the strongest reported reference is the metric-learning relabeling of Zhang et al., which reaches 0.67 sensitivity and 0.66 specificity on SCH-LND. We also run DiFF-style diagnosis-weighted fusion with a matched backbone.
The ablation that isolates the key ingredient keeps our full architecture but replaces each annotation's $c_r$ with ratings shuffled among raters of the same nodule. If the observed ratings carry rater-specific boundary information, the shuffled model should lose most of the concept-channel gain. A second ablation removes the residual $z$ entirely.
Beating the free-latent baseline is not required. Success means a test-set $E$ well above the shuffled-concept control with $z$ at a small budget. For diagnosis, success means SCH-LND specificity above 0.66 at sensitivity at least 0.67.
Why not just…?
Why not just train Pionono and then probe its latent for concepts afterwards? This is the most serious alternative and we run it. A linear or shallow probe from the free latent to $c_r$ can show correlation. However, the free latent may encode concepts entangled with unnamed factors, so a high probe score does not license editing a concept and trusting the resulting boundary. Our counterfactual test measures exactly that. If the probed free latent supports equally faithful edits and equal $E$, we will report that the simpler post-hoc route suffices and recommend it.
Why not just compute shape descriptors from each rater's mask and correlate them with that rater's ratings? This geometric analysis needs no deep model, and it may well answer the descriptive question of whether boundary differences track rating differences. We include it as the first analysis. It cannot, however, generate rater-specific boundaries for a new scan where no ratings exist. It also cannot attach a reason to a sampled contour, which is the deployable product. The proposal stands only if $p_\phi(c \mid x)$ makes this generation step useful.
Why not just collapse annotations by majority vote or STAPLE consensus and train a concept bottleneck? That is the naive baseline. Prior evidence argues against it. The Latent Doctor Model beat majority-vote training in matching expert label distributions, and UGS-Net beat single-annotation segmenters by learning from intersections and unions. Our experiment adds a sharper test: whether consensus-trained concept predictions are stable when the input mask is swapped among the four real rater masks.
Why not just weight raters by diagnostic usefulness as DiFF does? DiFF reported AUC gains of up to 2.73% on glaucoma, 2.51% on thyroid cancer, and 2.04% on melanoma. On LIDC, its diagnostic target would be biased radiologist scores. Its weights also explain nothing clinically. We compare against it on biopsy-confirmed SCH-LND, which is the comparison that can actually separate the two approaches.
Proposed Experiments
Rater-mask reproduction on LIDC-IDRI. We use all nodules with at least two annotations under 5-fold cross-validation and report $E$ alongside two distribution-level measures: generalized energy distance and Hungarian-matched IoU between sampled and real mask sets. We also report staged Dice in the QUBIQ style.
Decisive check: does the explanation hold up? The result counts if all three of the following hold:
- Shuffling concepts within a nodule collapses $E$.
- The edited-concept displacement maps overlap real disagreement zones significantly more than random concept edits do.
- For held-out rater pairs, the predicted mask difference from their concept difference correlates with their actual mask difference.
Failure of the first condition alone would mean that concepts and boundaries are not jointly informative at the rater level. That would be a negative result about the clinical vocabulary itself.
Robustness of concept models to boundary choice. We feed each of the four real rater masks into a consensus-trained PN-SAMP-style attribute predictor and into a mask-pooled phenotype classifier, and measure how often concept and malignancy predictions flip. We then compare with our model, which predicts a concept distribution instead of a point. This directly quantifies the brittleness left open by the lesion-phenotype work.
Diagnosis against pathology. We train the malignancy head on LIDC using concept samples, report sensitivity and specificity on SCH-LND against the 0.67/0.66 reference, and stratify by nodules on which raters disagreed in margin or spiculation.
Caveat. In LIDC, raters rated attributes after drawing. Concept–boundary co-variation may therefore reflect a shared perception rather than ratings causing contours. We claim explanation in the sense of shared, nameable variation, not causal direction, and the counterfactual analysis is framed accordingly.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- EDUE: Expert Disagreement-Guided One-Pass Uncertainty Estimation for Medical Image Segmentation
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Uncertainty-Guided Lung Nodule Segmentation with Feature-Aware Attention
- Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction
- Lesion-Centric Latent Phenotypes from Segmentation Encoders for Breast Ultrasound Interpretability
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Concept Bottleneck with Visual Concept Filtering for Explainable Medical Image Classification