Clinical diagnostic models are usually built to answer a yes/no question, such as "pulmonary embolism or not" or "melanoma or not." They take in a scan or a waveform, sometimes a table of lab values, and return a risk score that nobody can audit. Real diagnosis works differently. A clinician weighs several competing explanations at once. The evidence they use has names ("ST elevation," "raised troponin," "spiculated margin"). How much each piece of evidence counts depends on who the patient is: a bone lesion that looks identical on two radiographs means different things at age 12 and at age 60. We propose a diagnostic model whose only route from inputs to a diagnosis runs through named clinical findings, drawn from both the signal and the patient record. The links between those findings, and from findings to each candidate diagnosis, are a small graph whose edge weights are recomputed for every patient from their context, such as age, sex, and history. The model outputs a full differential, a probability over competing diagnoses, together with a per-patient map of which findings supported which diagnosis and how strongly.
The claim we want to test is narrow, and it can fail. Existing concept-based models either treat findings as independent, or learn one graph of relations between findings that is shared by every input and built only from images. Existing multimodal clinical models fuse imaging and records into opaque vectors, mostly for binary endpoints. We claim that making the finding-to-finding links both cross-modal and patient-conditioned does three things. It keeps accuracy at the level of black-box fusion on multi-class differential diagnosis. It makes clinician corrections and missing tests propagate sensibly, measurably better than a flat or globally fixed concept graph. And it removes the drop in accuracy on non-specific classes such as "normal" or "benign nevus" that concept supervision has been observed to cause.
Where this sits relative to prior work
A concept bottleneck model forces a network to first predict human-readable findings and then predict the label only from those findings. Several papers motivate the gap we target. Graph Concept Bottleneck Models learn a latent graph among concepts, trained with contrastive and sparsity terms, and pass messages over it. They report 1–2% average accuracy gains over flat concept bottlenecks across eight datasets, 77.95% versus 74.69% on CUB, and better behaviour when concepts are corrected by hand. As described, that graph is a single structure shared by all inputs, and its concepts come only from images. We keep its message-passing idea but add findings from other modalities and make the edges depend on the patient. Multimodal Concept Bottleneck Models align image and text concept layers but leave the concept vector flat, and they evaluate on natural-image benchmarks. Integrating Clinical Knowledge into Concept Bottleneck Models aligns concept reliance with expert importance tiers and improves out-of-domain F1 (43.64% to 49.75% on RaabinWBC), but it is unimodal and treats concepts as isolated. We borrow its perturbation-based reliance measure for our faithfulness checks.
On the graph side, Learning Differential Diagnosis of Skin Conditions with Co-occurrence Supervision using Graph Convolutional Networks shows that relating 80 disease classes through a graph helps differential diagnosis (top-5 accuracy 93.6% versus 91.8% for ResNet-101). Its graph, however, is fixed from training-set co-occurrence, links diseases directly to pixels, and ignores patient metadata. Predicting Alzheimer's Disease by Hierarchical Graph Convolution from Positron Emission Tomography Imaging similarly uses one cohort-wide brain graph. Heterogeneous Similarity Graph Neural Network on Electronic Health Records does learn an adaptive adjacency, by attention over meta-path subgraphs, and that fused adjacency counters the over-smoothing caused by hub nodes. It operates over patients and visits rather than over findings, uses no imaging, and offers no concept-level explanation. Together these papers show that relational structure helps and that static structure is a known weakness. None of them puts the adaptive structure at the level a clinician reads.
On the multimodal side, Impact of metadata in multimodal classification of bone tumours gives the clearest motivation. Age, added by concatenation, lifts 10-class accuracy from 62.8% to 69.7%, and SHAP identifies age as the dominant feature. Yet the visual branch stays opaque, and the rarest class gets 0% accuracy. CLIC: Contextual Language-Informed Cardiac Pathology Classification turns ECG metadata into text and fuses it with a ResNet18 embedding for 5-class PTB-XL diagnosis. PECon aligns CT and EHR features for binary embolism detection and explains them only with Grad-CAM. Dynamic Multimodal Information Bottleneck for Multimodality Classification compresses fused inputs into an unreadable vector. These are the accuracy competitors we must match.
Two papers constrain the design. Integrating Clinically-Relevant Features into Skin Lesion Classification found that supervising ABC criteria hurt benign-nevus performance. Any concept model must therefore be checked on non-specific classes, not only on the headline disease. Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations warns that explicit graph modeling is often redundant once representations are rich. We treat it as a live threat and test it directly rather than assume a graph helps.
Core Claim and Mechanism
Each patient contributes a vector of findings $c \in [0,1]^K$. Some findings are predicted from a raw signal by an encoder, for example QRS widening, ST deviation, or T-wave inversion from an ECG. Others are read directly from the record, for example troponin above the reference range or prior infarction. Each patient also contributes a small context vector $z$: age, sex, and a handful of history flags. The model is allowed to see only $c$ and $z$ after the encoders, with no residual channel from the raw embedding. This removes the "leakage" route that Multimodal Concept Bottleneck Models identify.
The model then does two things. In the first, a single round of message passing refines the findings over a sparse finding-to-finding graph whose edge weights depend on $z$. The refined findings are $\tilde{c} = c + \sigma(A(z)\,c)$, where $A_{ij}(z) = M_{ij}\,(a_{ij} + u_{ij}^{\top} z)$. Here $M$ is a learned sparse mask shared across patients, $a_{ij}$ is a base strength, $u_{ij}$ shifts that strength with context, and $\sigma$ is a squashing function. In the second, a score is computed for each of the $D$ candidate diagnoses, and a softmax over these scores gives the differential:
$$s_d = b_d + \sum_{k=1}^{K} W_{dk}(z)\,\tilde{c}_k, \qquad W_{dk}(z) = R_{dk}\,(w_{dk} + v_{dk}^{\top} z),$$
where $R$ is a sparse mask on finding-to-diagnosis edges, and $w_{dk}$ and $v_{dk}$ are the base and context-dependent evidence weights. For any patient, one can print the active edges and their weights, which yields a short, readable account. The account takes the form "for this 67-year-old man, ST elevation contributed +2.1 to infarction and pulled down the estimate of left ventricular hypertrophy through its link to voltage criteria."
The diagnosis head, viewed alone, is a sparse model of interactions between findings and context. We make that explicit because it fixes where the novelty can and cannot live. The head is a statistical model that already exists. What prior work lacks is the context-conditioned, cross-modal finding-to-finding layer. When a clinician corrects one finding, or when a test was never ordered, that layer decides how the other findings and the differential should move. The claim stands or falls on that layer.
Missing tests are handled without imputation of raw values. In the spirit of the "omit the node if the test wasn't ordered" design in Graph-Based Patient Representation for Multimodal Clinical Data, a missing lab finding is masked. Its value in $\tilde{c}$ comes only from messages sent by other findings. The resulting estimate is inspectable, so a clinician can see that "troponin was not measured; the ECG findings imply it is likely elevated." Training combines cross-entropy on the diagnosis, a concept loss on supervised findings, an L1 penalty on both masks, and, during training, random masking of whole modalities so that the propagation path is actually learned.
Research Program
The work has three parts. The first builds the model on top of fixed, shared encoders, so that differences between methods come from the finding-and-diagnosis layers only. The second defines finding vocabularies for each setting. These are fixed before training and drawn from clinical measurement definitions rather than learned, so that every node has a name a clinician would accept. The third evaluates along four axes that each correspond to a claim: differential accuracy, including tail classes; how efficiently clinician corrections improve the differential; robustness when a modality is missing; and accuracy on non-specific classes.
The binary-classification direction enters as an evaluation regime rather than as a mechanism. For each setting we also train the binary collapse that prior work used, and report what that collapse throws away. Specifically, we measure how often a patient the binary model calls "positive" is assigned the wrong positive subtype by the multi-class model, and whether the context-conditioned edges reduce those confusions.
The settings are chosen to isolate components. PTB-XL is the main setting, with 21,837 12-lead ECGs, demographic metadata, and five superclasses. We use the dataset's recommended patient-level folds. ECG findings such as interval durations, axis, ST and T morphology, and voltage criteria are computed with a fixed delineation tool. Diagnoses are evaluated both as CLIC's 5-class task and in the native multi-label form.
The second setting is MIMIC-IV-ECG, through MEETI, which supplies signals, extracted features, and interpretations. It is linked to MIMIC-IV labs and demographics, and this is where the cross-modal claim lives: ECG findings plus cardiac and inflammatory lab findings. Here the "test not ordered" condition occurs naturally rather than being simulated.
The third setting, LIDC-IDRI, is a unimodal control with nine radiologist attributes. Prior work collapsed it to benign versus malignant; we restore the 1–5 malignancy scale. It tests whether finding-to-finding edges help when there is no second modality, with context limited to nodule size and location.
The fourth setting, HAM10000, uses the age, sex, and lesion-site fields distributed with the dataset. It allows a direct comparison with Graph Concept Bottleneck Models on a dataset they report. It also covers the benign-nevus class on which the distraction effect was observed.
Baselines
The naive reference points are a majority-class predictor, a model that sees only the metadata, and the signal encoder with a linear head and no concepts or metadata. These establish how much of the task demographics alone explain. That matters because Impact of metadata in multimodal classification of bone tumours reports metadata-only accuracy of 18.5% against 62.8% for images. We also train one-vs-rest binary models per diagnosis, which represent the binary collapse common in the literature. Their combined argmax is the reference for "what you lose by asking yes/no questions."
The strongest accuracy competitor on PTB-XL is CLIC-style fusion. The ECG ResNet18 embedding is concatenated with a ClinicalBERT encoding of templated metadata text and fed to an MLP head, following CLIC. CLIC is reported to beat unimodal and attribute-concatenation baselines in accuracy and macro-F1, but no reported result was found on Lacuna for CLIC's exact PTB-XL numbers, so we reproduce it under our folds. On LIDC-IDRI the reference is PN-SAMP, which reports 97.58% binary malignancy accuracy and 89.33% attribute accuracy. Because we use the 5-level target, we reproduce PN-SAMP's binary setting first, to confirm our pipeline, and then retrain it on the ordinal target. Beating the black box means matching its macro-F1 within one standard deviation over five seeds while winning on the intervention and missing-modality axes. We do not claim to beat it on raw accuracy, and a loss of more than two macro-F1 points would count against the proposal.
The strongest interpretable competitor is a global-graph concept bottleneck. This is Graph Concept Bottleneck Models, extended so that lab and metadata findings are additional nodes in its single learned graph. Its reported 77.95% versus 74.69% on CUB shows that relational concepts help. The extension tests whether patient conditioning adds anything beyond putting all modalities into one shared graph. The flat alternative is a multimodal concept bottleneck with non-negative linear weights over all findings, in the style of Multimodal Concept Bottleneck Models. We also run a sparse interaction model that takes the same findings and context, with pairwise finding-by-finding and finding-by-context terms selected under the same sparsity budget.
Two ablations isolate the key ingredient. The first sets $u_{ij}=0$ and $v_{dk}=0$, which turns the model into a global graph with our exact architecture. The second removes the finding-to-finding layer, leaving only the context-conditioned head. All methods share frozen or identically fine-tuned encoders, the same finding vocabulary, the same patient-level splits, and the same number of hyperparameter trials. We use 40 random-search configurations each, selected on validation macro-F1, and report means over five seeds.
Why not just…?
Why not just use black-box multimodal fusion and explain it with SHAP? This is what Impact of metadata in multimodal classification of bone tumours and PECon (with Grad-CAM) do. It may well win on raw accuracy. Its explanations, however, are post hoc and cannot be acted on: a clinician cannot correct a SHAP value and see the differential update. Our intervention experiment answers this directly. A post-hoc explainer has no intervention curve at all. We also run a deletion test: the edges our model claims matter are removed, and the resulting drop in prediction is compared with removing the features SHAP ranks highest in the black box. That tells us whether either explanation is faithful. If the black box is both more accurate by a wide margin and its SHAP attributions pass the deletion test as well as ours, the interpretability case weakens, and we will report that.
Why not just use a flat concept bottleneck with a linear head over all findings, including labs and age? It is simpler and fully readable. But a linear head cannot express "this finding matters more at age 60 than at 12," which is exactly the age effect the bone-tumour study surfaced. Flat bottlenecks also cannot move one finding when another is corrected. Graph Concept Bottleneck Models report that relational concepts improve intervention gains, especially when few concepts are corrected. The flat baseline is run in every setting. If it matches us on intervention and missing-modality curves, the graph is unnecessary.
Why not just fit a sparse interaction model on findings and context, with no graph? This is the most serious threat. Our diagnosis head is mathematically such a model, so on fully observed inputs the two could tie, and we expect they might. The difference shows up only when findings change after prediction or are absent. An interaction model has no mechanism for inferring a missing troponin from ECG findings, or for propagating a corrected ST finding to related findings. The MIMIC-IV setting, where tests are genuinely unordered, and the intervention curves are where this is settled. If the interaction model matches us there too, for example by mean-imputing missing findings, then the finding-to-finding layer adds nothing, and we would narrow the contribution to an evaluation study.
Why not just serialize everything into text for a language model with an MLP head? Do We Really Need GNNs with Explicit Structural Modeling? argues that MLPs over rich language-model representations often make graphs redundant, and CLIC shows templated text works for ECG metadata. We include an MLP over a ClinicalBERT encoding of serialized findings plus metadata. This baseline may match us on accuracy. What it cannot provide is a per-patient, editable evidence map, and that is the property under test, not accuracy alone.
Proposed Experiments and Decisive Checks
The first check concerns accuracy and the differential. On PTB-XL and HAM10000 we report macro-F1, per-class recall, with emphasis on the rarest classes, and confusion between positive subtypes, against all baselines. We also measure how often the binary-collapse models confuse which positive subtype a patient has.
The second check concerns intervention. Following the protocol in Graph Concept Bottleneck Models, we replace predicted findings with ground truth in order of model uncertainty, at ratios from 0 to 50%, and plot macro-F1 against the number of corrections. The decisive comparison is our model against the global-graph ablation at low correction ratios. If patient conditioning gives no gain there, we drop that claim.
The third check concerns missing modalities. In MIMIC-IV we evaluate separately on patients who lack specific labs, and on PTB-XL we mask finding groups at random. We report accuracy and also the calibration of propagated finding estimates against held-out true values.
The fourth check concerns distraction. We compare accuracy on NORM in PTB-XL and on nevus in HAM10000 between the concept-free encoder and each concept model. This tests the hypothesis that relational, context-gated findings avoid the degradation reported by Integrating Clinically-Relevant Features into Skin Lesion Classification. A drop of more than one point on these classes would show that the problem persists.
The fifth check concerns faithfulness and stability. We remove the top-ranked edges, measure the change in prediction using the perturbation reliance measure of Integrating Clinical Knowledge into Concept Bottleneck Models, and measure the overlap of active edge sets across seeds. An explanation that changes when only the random seed changes is not a clinical explanation.
On LIDC-IDRI we additionally test whether finding-to-finding edges improve attribute accuracy over PN-SAMP's independent attribute heads, using its reported 89.33% as the reference point. This isolates the value of the relational layer when no second modality is present.
References
- Graph Concept Bottleneck Models
- Multimodal Concept Bottleneck Models
- Integrating Clinical Knowledge into Concept Bottleneck Models
- Learning Differential Diagnosis of Skin Conditions with Co-occurrence Supervision Using Graph Convolutional Networks
- Predicting Alzheimer's Disease by Hierarchical Graph Convolution from Positron Emission Tomography Imaging
- Heterogeneous Similarity Graph Neural Network on Electronic Health Records
- Impact of metadata in multimodal classification of bone tumours
- CLIC: Contextual Language-Informed Cardiac Pathology Classification
- PECon: Contrastive Pretraining to Enhance Feature Alignment Between CT and EHR Data for Improved Pulmonary Embolism Diagnosis
- Dynamic Multimodal Information Bottleneck for Multimodality Classification
- Integrating Clinically-Relevant Features into Skin Lesion Classification
- Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations
- Graph-Based Patient Representation for Multimodal Clinical Data: Addressing Data Heterogeneity
- MEETI: A Multimodal ECG Dataset from MIMIC-IV-ECG with Signals, Images, Features and Interpretations
- Joint learning for pulmonary nodule segmentation, attributes and malignancy prediction