Most multimodal medical models ask whether combining images with clinical text improves a yes/no diagnosis. For decisions that depend on how severe a cancer is, a more useful question is narrower: at which rungs of a graded scale does quantitative image analysis add information beyond what the radiologist already wrote in the report? We propose to answer this for prostate cancer, where the target is the biopsy-defined grade (the ISUP grade group, a five-level pathology scale). Two input sources compete. The first is structured findings pulled from narrative MRI reports and clinical notes by a verifiable language-model extraction pipeline. The second is radiomics, meaning handcrafted numerical descriptors of lesion shape, intensity distribution, and texture computed from the scan. We attribute the model's predictions to these two sources, and to families of radiomic features within the second, separately for each grade boundary. The attribution uses SHAP, an explanation method from cooperative game theory that splits a prediction into additive contributions from inputs. Every attribution comes with a stability audit, so that a claim like "texture matters at the ambiguous middle grades" can be shown to survive re-running the explainer and retraining the model.
The motivation is a pattern in the literature. Where the diagnostic tier is itself defined by descriptive terms that appear in reports, text wins trivially. In thyroid staging, embeddings of structured TI-RADS descriptor templates alone reach 98% accuracy, against 75.17% balanced accuracy for the best image model (CMCNet). That result says little about biology, because the tier is computed from those descriptors. Where the endpoint is pathology, the advantage of adding clinical context can disappear on external data. The prostate foundation model MFM-Geom gains strongly from clinical text in low-data training, yet on external PROSTATE158 its multimodal and image-only variants score essentially the same AUC-PR (90.6 vs 90.61). A clean, stable, tier-by-tier account of incremental image evidence is therefore both missing and necessary before anyone builds granular multimodal tools on top of reports.
Core Claim
Against a pathology-defined granular endpoint, radiomic features carry information beyond the extracted report content. We hypothesize that this information is concentrated at intermediate boundaries, chiefly ISUP 1 versus 2 among lesions the report scores as equivocal (PI-RADS 3), rather than at the extremes, where report descriptors already suffice. We further claim that this tier-dependence is measurable reliably only at the level of feature families (shape, first-order intensity, co-occurrence and run-length texture, wavelet and Laplacian-of-Gaussian filtered texture). Per-feature SHAP rankings over hundreds of collinear radiomic features will show the rank churn documented by Explanation Multiplicity in SHAP. The claim can fail in two ways. If incremental value is flat across tiers, the hypothesis is refuted. If family-level attributions are as unstable as per-feature ones, the explanation component adds nothing.
A secondary, practical claim is that the per-patient radiomic attribution flags cases where the report and the pathology disagree. The strongest version of this claim is that it flags them better than simply checking whether a report-only model and a combined model disagree.
Why the Obvious Version Fails
The obvious project concatenates LLM-extracted report fields with radiomics, trains a multi-class grader, and prints a SHAP summary plot. The GIST Ki-67 study does most of this, minus the text extraction. It extracts 1,316 radiomic features, compresses them into a Radscore, combines them with radiologist-recorded semantic features, and uses SHAP to name tumor diameter and Radscore as the main drivers. That design has two problems here.
The first is leakage. Reports contain the radiologist's final assessment category and phrases such as "suspicious for high-grade disease." A model given these will attribute almost everything to text for reasons that have nothing to do with the question. The CMCNet 98% figure is the warning. Our design therefore treats report content in explicit layers: descriptors only; descriptors plus size and location; and descriptors plus the final PI-RADS category. The main analysis conditions on the full report, so incremental radiomic value is measured against everything the radiologist committed to writing.
The second is that per-feature SHAP over correlated texture features does not give reproducible rankings. Towards Trustable SHAP Scores shows that the standard conditional-expectation formulation can give nonzero credit to irrelevant features and misorder features when they interact. The multiplicity paper shows that stability metrics based on magnitude hide top-k rank churn even in confident predictions. Both results constrain the design. Neither was tested on radiomics, and radiomics is the harder case.
Research Program
The pipeline has three components, each reusing an existing method.
Text extraction. For each MRI exam, we extract structured fields from the narrative report and the preceding clinical notes. The report fields are lesion zone, size, T2 and diffusion descriptors, extraprostatic extension language, and the assessment category. The note fields are PSA values and dates, prostate volume, prior biopsy results, and family history. Extraction follows the four-step pipeline of Self-Verification, which runs an initial extraction, a search for omissions, verbatim evidence quoting, and pruning of negated or outdated items. Before any value is extracted, the presence-checking step of mCODEGPT's Two-Phase Ontology Parser confirms each field is actually in the text. These methods address the hallucination failure that would otherwise inject fabricated grades or PSA values into the grader. Every extracted value keeps its evidence span, so downstream attributions trace back to a sentence.
Radiomics. For each index lesion, we compute radiomic features on T2-weighted, diffusion, and ADC sequences using the standard hierarchy of first-order, shape, co-occurrence, run-length, and filtered texture features. We filter features for reproducibility by intraclass correlation, as in the GIST study, and assign each feature to one of five families. Multi-vendor variation is a known weakness of handcrafted features, so scanner vendor is recorded and used for held-out-vendor splits.
Grading and attribution. The grader is an ordinal model: gradient-boosted trees with cumulative-link outputs, plus a penalized ordinal logistic regression as a check. We train at cohort sizes typical of radiomics (a few hundred lesions) and deliberately keep the model small. Attribution is computed at the group level, at each boundary $j$ between adjacent grades, on the cumulative log-odds output $f_j(x)$ of the ordinal model. With groups $G$ (report fields, clinical-note fields, and the five radiomic families), the contribution of group $g$ for patient $x$ is
$$\phi_g^{(j)}(x)=\sum_{S\subseteq G\setminus\{g\}}\frac{|S|!\,(|G|-|S|-1)!}{|G|!}\Big[v_j(S\cup\{g\})-v_j(S)\Big],$$
Here $v_j(S)$ is the boundary-$j$ output when only the groups in $S$ are taken from $x$ and the rest are drawn from a background set of patients. With seven groups there are 128 coalitions, so the sum is computed exactly, which removes Monte Carlo explainer noise altogether. The remaining instability comes from the model and the background set, and we measure it with the dual-seed protocol of the multiplicity paper, which varies the training seed and the explainer seed independently. We report top-k Jaccard and rank-biased overlap against that paper's randomized null rankings.
Baselines
The naive reference point is the report itself. An ordinal model uses only the extracted report and note fields, and a simpler rule maps the PI-RADS category to predicted grade. This represents doing nothing beyond what the radiologist wrote. Any gain from radiomics must beat it within each tier, not just in aggregate.
The strongest existing method for combining prostate MRI with clinical information is MFM-Geom. It feeds a BiomedCLIP-derived 3D image encoder and PubMedBERT-encoded clinical templates into a Riemannian classification head, and reports AUC-PR 90.67 on PI-CAI when trained on 10% of the data (binary clinically significant cancer). We will retrain it on our splits with its structured clinical inputs replaced by our extracted fields. Its binary head becomes an ordinal head, and it gets the same hyperparameter-search budget as our grader. It is the strongest competitor because it tests whether dense learned image features make handcrafted radiomics unnecessary. Beating it means matching or exceeding its quadratic-weighted kappa on grade while showing tier-resolved incremental value that MFM-Geom's opaque fusion cannot report. If it clearly wins on accuracy, we report that and fall back to measuring incremental value of its image embedding by the same group decomposition.
For the guideline-distillation approach of the PI-RADS MLLM paper, the file reports only relative accuracy gains of 4.8–6.4% on its private cohort. We find no absolute number on Lacuna and no reported result on Lacuna for ISUP grade-group prediction from combined report and radiomic inputs, so our report-only baseline is the anchor.
Three ablations isolate the key ingredients. The first swaps verified extraction for single-pass prompting and for manually abstracted gold fields, to measure how extraction errors propagate into grade accuracy and attributions. The second computes per-feature SHAP on the same model to quantify how much of the stability comes from grouping. The third reruns everything with the final assessment category removed from the report features, to measure leakage directly.
Why not just…?
Why not just compare report-only and report-plus-radiomics models tier by tier, without any attribution? This is the most serious alternative. For the population-level question it answers most of what we ask, and we will run it first as the primary incremental-value test: per-boundary AUC difference with bootstrap intervals. Attribution earns its place only in two roles. The first is splitting the radiomic gain across feature families, which modality ablation cannot do without training 2^5 models on a few hundred lesions. The second is producing per-patient signals. If the family-level decomposition merely reproduces the ablation ranking and adds nothing at the patient level, we will say so.
Why not just flag patients where the report-only and combined models disagree? For two groups, the interventional group Shapley value of radiomics is half the disagreement term plus half the standalone radiomic contribution, so the two signals are closely related. The discordance experiment below compares them directly on the same patients. If disagreement flags report–pathology discordance as well as family-level attribution does, the simpler signal should be deployed. Our remaining contribution would then be the leakage-controlled, tier-resolved measurement.
Why not just use default per-feature TreeSHAP and read the top features? Defaults are cheap, and TreeSHAP on trees has no sampling noise. The multiplicity paper shows, however, that model-induced variation alone produces rank churn. Radiomics sharpens the problem because tree models split arbitrarily among near-duplicate wavelet textures. The per-feature ablation measures this: under the dual-seed protocol, we expect per-feature top-10 overlap to sit near the Mallows null while family-level rankings sit well above it. If per-feature rankings prove stable, grouping is unnecessary and we drop it.
Why not just use dense foundation-model image features, which already beat handcrafted features in CMCNet and MFM-Geom? They may predict better. But the question is what kind of image evidence goes beyond the report, and a 768-dimensional embedding cannot answer it. The MFM-Geom baseline above settles the accuracy side. Our program reports both.
Proposed Experiments
The main cohort must pair narrative MRI reports and clinical notes with biopsy ISUP grades. Public data in the file do not provide narrative reports. We therefore need an institutional cohort, ideally multi-vendor like the Prince of Wales cohort (293 lesions, 206 patients) described for the PI-RADS MLLM work, with a hand-abstracted subset of about 150 reports to score extraction accuracy. Two public datasets serve as a structured-inputs replication: PI-CAI, with age, PSA, and volume as clinical fields, and Prostate-MRI-US-Biopsy, with targeted biopsy outcomes. Grade labels will be used where the releases provide them. We test on PROSTATE158 externally, given the parity between image-only and multimodal models reported there. The thyroid STN dataset (600 nodules) serves as a deliberate leakage control. Because TI-RADS tiers are defined by lexicon descriptors, the method should attribute nearly everything to descriptors there. If it does not, the attribution is broken.
The decisive checks are four. The first is whether the report-conditioned radiomic gain in AUC at the ISUP 1/2 boundary among PI-RADS 3 lesions exceeds the gain at the extreme boundaries, with non-overlapping bootstrap intervals, on the internal test set and under held-out-vendor splits. The second is whether family-level attributions reach rank-biased overlap clearly above the randomized null under the dual-seed protocol while per-feature attributions do not. The third is whether substituting single-pass extraction for verified extraction measurably shifts grade accuracy and group attributions. The Self-Verification paper reports ICD-10 F1 moving from 0.487 to 0.533, and mCODEGPT reports 93.9% accuracy with 1.1% hallucination under presence checking, so extraction errors of that size are plausible. The fourth is the discordance test. We define report–pathology discordance as a PI-RADS 3 lesion with ISUP ≥ 2, or a PI-RADS 4–5 lesion that is benign or ISUP 1. We then ask whether the per-patient radiomic group attribution identifies discordant cases, measured by AUC, better than both report-versus-combined model disagreement and the combined model's predictive entropy. A positive result on the first and second checks establishes the scientific claim. A positive result on the fourth justifies building a review-routing tool. A negative result on the fourth with positive results on the first two means the simpler disagreement check should be used in practice.
References
- CMCNet: Aligning Ultrasound Image Embeddings with Textual TI-RADS Representations for Fine-Grained Thyroid Classification
- A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification
- Explanation Multiplicity in SHAP: Characterization and Assessment
- Interpretable machine learning model based on CT semantic features and radiomics features to preoperatively predict Ki-67 expression in gastrointestinal stromal tumors
- Towards Trustable SHAP Scores
- Self-verification improves few-shot clinical information extraction
- Introducing mCODEGPT as a zero-shot information extraction from clinical free text data tool for cancer research
- Incorporating Clinical Guidelines Through Adapting Multi-modal Large Language Model for Prostate Cancer PI-RADS Scoring