Clinical prediction models built from small, multi-center cohorts usually pass their candidate features through a funnel. The first step drops features that change when a different radiologist outlines the lesion. The second drops features that duplicate one another. The last fits a sparse regression that keeps a handful. Every step in this funnel is scored against one label per patient, and that label is often a majority vote that hides real disagreement among readers. We propose to change what the funnel selects for. Each patient's target becomes two things: the diagnosis, and how much the experts disagreed about it. The funnel keeps features that predict either one. The final sparse model then outputs a full distribution over expert opinions instead of a single probability. Imaging and clinical variables, such as age or serum markers, compete inside the same funnel. Clinical variables are not added afterwards as a separate covariate.
The bet is that the variables explaining where experts split differ from those explaining what the diagnosis is. We also expect patient context to matter most for the first. A radiologist looking at a borderline cyst on MRI does not see the patient's age or tumor markers, yet those may be what make the case ambiguous or resolve it. If so, a disagreement-aware funnel should yield models that are small, better calibrated, and more stable across hospitals. It should also be able to flag the cases on which human readers are likely to be wrong.
Core Claim
A three-stage selection funnel can be built in which features are scored jointly against the outcome and against rater disagreement, measured as the entropy of the readers' labels. Such a funnel produces a sparse multimodal model with two parts. One part predicts the diagnosis. The other predicts how dispersed expert opinion will be. We claim this model beats two kinds of competitor at matched feature budgets, in calibration and in agreement with the human disagreement pattern, while matching their discrimination on unseen centers:
- funnels trained on consensus labels and then recalibrated;
- end-to-end soft-label networks that use no selection.
The new element is not the funnel, and it is not soft labels. It is using disagreement as a selection target, so that the selected feature set itself becomes an interpretable account of diagnostic ambiguity.
Where This Sits
The funnel itself is established. T2-Weighted Image-Based Radiomics Signature for Discriminating Between Seminomas and Nonseminoma filters 851 MRI features by intra- and inter-observer ICC, then by mRMR, then by LASSO, ending with five features in 39 patients. ICC (intraclass correlation) measures how consistent a feature is across repeated outlines; mRMR (maximum relevance, minimum redundancy) keeps features informative about the label but not about each other. Radiomics-based infarct features on CT predict hemorrhagic transformation in patients with acute ischemic stroke uses ICC > 0.75 and univariate tests before LASSO, and only afterwards compares the resulting score with clinical scores in a separate regression. Both supply our method. Both also illustrate the two habits we break: clinical data sit outside the funnel, and observer variability is treated purely as contour noise to be removed, never as a quantity to be predicted.
Inter-modality Relationship Constrained Multi-Task Feature Selection for AD/MCI Classification does select jointly across MRI and PET. It preserves distances between subjects across the two modalities, but it assumes crisp diagnoses and covers no tabular clinical data. We borrow its multi-task, group-sparse structure. Our "tasks" are the diagnosis and the disagreement, not the modalities.
On the disagreement side, three papers motivate the proposal by showing that collapsing raters harms models:
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel trains one network per pathologist. The resulting panel reaches F1 78.2% and ECE 0.050, against 72.9% and 0.183 for consensus training.
- The Latent Doctor Model for Modeling Inter-Observer Variability models the rater distribution with a conditional variational autoencoder. Its predictive entropy reaches Spearman correlation up to 0.728 with 20 pathologists.
- Calibrate the inter-observer segmentation uncertainty via diagnosis-first principle reweights rater masks by how much they help diagnosis.
All three are image-only and dense. None tells a clinician which measurable patient properties drive the disagreement. Our proposal targets exactly that, using a few selected variables.
On the multimodal side, two papers show that context changes diagnoses. Impact of metadata in multimodal classification of bone tumours raises accuracy from 62.8% to 69.7% by adding age and site. ProstNFound gains over 7% AUROC from clinical prompts. Dense fusion models such as MFM-Geom feed all tokens forward with no pruning. Our application regime comes from Multi-center evaluation of radiomics and deep learning to stratify malignancy risk of IPMNs. That study covers 359 MRI scans from seven centers, read by three radiologists whose accuracies range from 37.4% to 66.7%. The authors name the missing clinical parameters as a limitation, and they report that 3D radiomics overfit across centers. That last finding constrains us: we start from the 2D feature set they found robust.
Research Program
Stage 1: stability. Imaging features are kept if their ICC across available rater segmentations exceeds 0.75, following the stroke study. Clinical variables cannot be tested that way. Instead they pass a missingness filter and a cross-center consistency check, and any variable whose distribution is mostly explained by the hospital it came from is dropped.
Stage 2: redundancy, with two targets. mRMR is run twice:
- once with relevance measured as mutual information with the reference diagnosis (histopathology where available, otherwise the rater mean);
- once with relevance measured as mutual information with per-case rater entropy.
The union of the two lists, tagged by origin, moves on.
Stage 3: a sparse two-headed model. It predicts a Dirichlet distribution over the $K$ diagnostic classes. A Dirichlet is a distribution over probability vectors, so it can express both which class is likely and how confident readers would be. Its parameters are written as $\alpha = s \cdot \pi$, where:
- $\pi = \mathrm{softmax}(W_\pi x)$ is the expected label distribution;
- $s = \exp(w_s^\top x)$ is a concentration, where low values mean high predicted disagreement;
- $x$ is the selected multimodal feature vector.
Training maximizes the Dirichlet likelihood of the observed rater votes. When histopathology exists, a cross-entropy term on $\pi$ against it is added. A group penalty on each feature's pair of coefficients (its row of $W_\pi$ together with its entry in $w_s$) lets a feature enter the diagnosis head, the disagreement head, or both. The fitted coefficients can therefore be read directly as "drives diagnosis" or "drives ambiguity."
All three stages run strictly inside each training fold. This avoids the leakage that inflates many small-cohort radiomics results.
Data. The primary testbed is the seven-center IPMN cohort. Its three radiologist reads plus histopathology give both a disagreement signal and an independent truth. The central dependency is obtaining the clinical variables (age, serum markers) that its authors flag as missing. Three raters give a coarse disagreement signal, so we add DEPP, the Dutch Esophageal Pathology Panel, with about 11 pathologists per case, provided basic patient metadata can be linked. On DEPP, image features are attention-pooled whole-slide features rather than radiomics, and Stage 1 uses stability across tile resampling instead of contour ICC. If no metadata can be linked for DEPP, it tests only the disagreement half of the claim, and we say so.
Baselines
The naive reference is the human reader. On the IPMN cohort, the radiologists' majority vote reaches 66.9% accuracy against pathology (Bejar et al.). A second naive model is the conventional funnel (ICC, mRMR, LASSO) fit to the majority-vote label using imaging only, which reproduces the radiomics arm of that study.
The strongest existing method on this cohort is the radiomics-plus-DenseNet decision fusion of Bejar et al.. It reports 74.3% AUC in cross-validation and 69.2% AUC on unseen hospitals. We re-run it on identical folds, with leave-one-center-out splits, under the same hyperparameter search budget as our model. Beating it means two things. First, the external AUC must be at least its reported 69.2% and not significantly lower on our own re-run. Second, ECE (expected calibration error) and the correlation between predicted and observed rater entropy must be clearly better, with at most 20 selected features.
On DEPP, the strongest disagreement-aware competitor is the virtual expert panel, with a reported ECE of 0.050 and F1 of 78.2% (More is more). We also train the Latent Doctor Model on the same features. For reference, it reports a KL divergence of 0.461 on Gleason2019 (LDM). No reported result was found on Lacuna for it on DEPP or on the IPMN cohort. Here we do not expect to beat the panel's ECE outright. Success means coming within its confidence interval while using one sparse model with tens of interpretable features, instead of 11 deep networks.
Three ablations isolate the key ingredient:
- the full funnel and Dirichlet model without the disagreement relevance target in Stage 2;
- the same model with clinical variables added only after selection, as in the stroke study;
- the same model with the disagreement head removed.
Why not just…?
Why not just train an end-to-end multimodal network on soft labels and skip feature selection? This is a real threat for discrimination. It would show, for example, whether DenseNet image features plus an MLP over clinical data, trained on rater-vote distributions, simply wins. The multi-center IPMN result argues against unconstrained capacity on 359 scans, since higher-dimensional 3D features overfit across centers. Still, we run this model on the same leave-one-center-out folds. If it matches us on AUC, calibration, and entropy correlation, our remaining contribution is interpretability and selection stability across bootstrap resamples, which we measure with Jaccard overlap. We would report that honestly rather than claim accuracy gains.
Why not just use the standard consensus-label funnel and fix calibration with temperature scaling afterwards? Temperature scaling can repair average confidence. It cannot make confidence vary case by case with which patients are ambiguous, because it rescales every prediction by the same factor. The decisive test is per-case: Spearman correlation between predicted dispersion and observed rater entropy, and selective-prediction curves against pathology. If consensus training plus temperature scaling matches us on those, the disagreement target is unnecessary.
Why not just fit a single-stage LASSO directly on soft labels over all multimodal features? With hundreds of collinear radiomic features and a few hundred patients, single-stage sparse selection is known to pick features arbitrarily across resamples. That is the reason the funnels in Land et al. and Zhang et al. pre-filter. We include this model with the same Dirichlet likelihood as a direct ablation and compare both performance and bootstrap selection stability. If it is as stable and as accurate, we drop Stages 1 and 2 and keep only the two-headed sparse model.
Decisive Checks
The hypothesis lives or dies on three checks.
- Do the two targets select different features? The diagnosis and disagreement lists from Stage 2 must overlap little (low Jaccard), and clinical variables must enter the disagreement head at higher rates than the diagnosis head. If the lists coincide, disagreement is just a monotone function of distance from the decision boundary, and the idea collapses into calibration.
- Does predicted disagreement flag human error? On cases where radiologists contradict pathology, the predicted concentration should be lower. Deferring the most-ambiguous predicted cases should raise accuracy on the rest, both for the model and for the reader majority vote.
- Does it hold up across hospitals? In leave-one-center-out evaluation, the disagreement-aware model should hold external AUC at or above the fusion baseline's 69.2% with far fewer features, and its selected set should stay stable across held-out centers.
Passing the first two with an unchanged third would already be a useful finding. It would mean diagnostic ambiguity can be measured and explained from patient data, and a sparse clinical model can be asked not just "what is it" but "would experts agree."
References
- T2-Weighted Image-Based Radiomics Signature for Discriminating Between Seminomas and Nonseminoma
- Radiomics-based infarct features on CT predict hemorrhagic transformation in patients with acute ischemic stroke
- Inter-modality Relationship Constrained Multi-Task Feature Selection for AD/MCI Classification
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Impact of metadata in multimodal classification of bone tumours
- ProstNFound: Integrating Foundation Models with Ultrasound Domain Knowledge and Clinical Context for Robust Prostate Cancer Detection
- A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification
- Multi-center evaluation of radiomics and deep learning to stratify malignancy risk of IPMNs
- Kernelized partial least squares for feature reduction and classification of gene microarray data