Checking Concepts Instead of Waiting for Outcomes: An Early Warning for Deployed Diagnostic Models

A diagnostic model trained on historical hospital data can report excellent accuracy and then fail on the uncurated patients of a live clinic. Nobody finds out quickly, because the ground truth it predicts, such as malignancy, treatment response or a cardiac event, arrives months or years later. The intermediate findings a clinician reads off a scan are different. Whether a lung nodule's margin is spiculated, whether a skin lesion is asymmetric, whether a white blood cell nucleus is lobulated: a clinician can check these on the day of the scan. We propose to use those findings as a fast audit. The model is forced to predict them and to base its diagnosis only on them, and a few clinicians spot-check them on incoming cases. The rate of disagreement then serves as a same-day estimate of how much the model's diagnostic accuracy has degraded, long before outcome labels exist.

The argument rests on one structural fact. In a concept bottleneck model, a network whose final diagnosis is computed only from its predicted clinical findings, a spurious shortcut in the new data can hurt the diagnosis only by corrupting the findings first. The findings are exactly what clinicians can verify. An ordinary black-box network with an extra findings head attached on the side offers no such guarantee. It can describe spiculation correctly while basing its malignancy call on a scanner watermark. The proposal therefore has two parts. One is a monitoring method for the gap between retrospective validation and prospective deployment. The other is a direct test of whether that monitoring ability is worth the accuracy that bottlenecks sometimes cost.

Core Claim

Under shifts between a retrospective training archive and a deployment stream, the error rate of a concept bottleneck model's findings, as judged by clinicians, predicts its downstream diagnostic accuracy substantially better than three alternatives. The first is label-free confidence or input-drift monitors. The second is findings heads attached to a black-box model. The third is waiting for an equally sized batch of outcome labels, judged per unit of time and annotation cost. The advantage should be largest for exactly the failures that defeat retrospective validation: shortcuts tied to acquisition site and artifacts.

Write $c$ for the vector of clinical findings, $\hat c$ for the model's predicted findings, $\hat y = g(\hat c)$ for the diagnosis computed from them, and $y$ for the eventual outcome. For a hard bottleneck the deployment error splits as

$$P(\hat y \neq y) = \sum_{\hat c} P(\hat c)\, P(g(\hat c) \neq y \mid \hat c).$$

The first factor depends on how inputs map to findings. This is where acquisition shortcuts and artifacts act, and clinician spot-checks estimate it. The second factor depends only on how findings relate to outcomes. It can drift with treatment standards, as the glioblastoma and lymphoma signature literature warns (Novel tumor necrosis factor-related long non-coding RNAs signature for risk stratification and prognosis in glioblastoma; A prognostic 15-gene model based on differentially expressed genes among metabolic subtypes in diffuse large B-cell lymphoma). It is not affected by changes in the images themselves. The monitor is therefore designed to catch input-side shift and is expected to miss outcome-side shift. We test both cases deliberately rather than claim coverage of everything.

Where This Sits

Prior work gives three kinds of evidence. Some of it motivates the gap. The prospective fetal ultrasound study (Clinical validation of explainable AI for fetal growth scans through multi-level, cross-institutional prospective end-user evaluation) deployed a progressive concept bottleneck on live videos and had 46 clinicians rate its explanations. It treated the concepts as something to show users, not as a measurement of model health. VentNet (Development and Prospective Validation of a Transparent Deep Learning Algorithm for Predicting Need for Mechanical Ventilation) validated prospectively, but it had to wait for intubation outcomes and explained itself with post-hoc gradients. The intravascular imaging model of Automated diagnosis of optical coherence tomography imaging on plaque vulnerability and its relation to clinical outcomes in coronary artery disease linked predictions to events over a median 530 days. Its authors concede that the retrospective cohort left out borderline and technically compromised pullbacks. That delay, and that blind spot, are what we target.

Other work supplies the method. PN-SAMP (Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction) shows that the nine LIDC-IDRI radiologist attributes can be predicted jointly with malignancy. The clinical alignment loss of Integrating Clinical Knowledge into Concept Bottleneck Models gives concept bottlenecks with expert-ranked concept importance on white blood cell and dermatology data, with defined cross-center shifts. We reuse both architectures, but we ask a different question. They ask whether concepts make the model more robust. We ask whether concepts make its loss of robustness visible early.

The rest constrains the design. Integrating Clinically-Relevant Features into Skin Lesion Classification found that forcing ABC criteria lowered accuracy on benign nevi, a "distraction effect". A monitor is worthless if the monitored model is noticeably worse, so the accuracy cost of the bottleneck is measured as a first-class outcome. Detecting shortcut learning for fair medical AI using shortcut testing showed that encoding an attribute is not the same as using it. That is exactly why findings heads on a black-box are suspect monitors: correct encoding of findings does not show that the diagnosis uses them. Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teachers supplies the synthetic shortcut protocol (diffuse noise, constant squares, random squares) and the CheXpert-to-MIMIC transfer that we use to create controlled failures.

Research Program

We build three model families on matched backbones. The first is a hard concept bottleneck, in which the diagnosis is a small head over predicted findings only, optionally trained with the clinical alignment loss. The second is a black-box classifier with auxiliary findings heads that share its features, in the style of PN-SAMP's multi-task design. The third is a plain black-box classifier.

Each model is trained on a retrospective source domain and scored on a stream from a shifted target domain, released in ordered batches as a prospective emulation. For each batch we compute several monitor signals. The concept monitor is the findings error rate on a randomly sampled audit subset of $k$ cases, judged against held-out expert findings. It is mapped to predicted diagnostic accuracy through a calibration curve fitted only on source validation data and on synthetic perturbations of it, never on target labels. The label-free monitors are mean confidence, predictive entropy, and an input-embedding two-sample drift statistic. The outcome-sampling reference uses $k$ true diagnosis labels from the same batch.

Every monitor is judged on four things. The first is how well its estimate tracks true diagnostic accuracy per batch, measured by mean absolute error and by rank correlation across shift conditions. The second is how many audited cases it needs to raise an alarm for a 10-point accuracy drop at a fixed false-alarm rate. The third is how often it misses failures. The fourth is the bottleneck's in-domain and out-of-domain accuracy relative to the black box.

The shift conditions are chosen so that the decomposition above makes falsifiable predictions. The datasets come from the file. White blood cell classification trains on PBC and streams to RaabinWBC and Scirep, which are natural staining and hardware shifts. Dermatology trains on Fitzpatrick 17k and streams to DDI, which shifts toward darker skin tones. Lung nodules come from LIDC-IDRI, where we inject synthetic shortcuts that correlate with malignancy in training and are decorrelated in the stream. Separately, on LIDC-IDRI we induce outcome-side shift by resampling the stream so that the attribute-to-malignancy relationship changes while images are untouched. The concept monitor should catch the first three conditions and miss the last. Chest X-ray transfer from CheXpert to MIMIC with pneumothorax shortcuts serves as a high-stakes stress test. No findings labels exist there, so only the label-free monitors and outcome sampling run, which bounds how far our claims reach.

If a clinical partner allows it, a final silent-mode arm runs a LIDC-trained bottleneck on newly acquired chest CT. Radiologists score the nine attributes on a random audit sample at the time of reading, and malignancy is confirmed later. This is the only arm that touches real prospective data. The emulated arms are designed so that a negative result there would stop the work before any clinical cost is incurred.

Baselines

The naive reference is no monitoring at all. It assumes deployment accuracy equals retrospective validation accuracy, which is exactly the assumption the bias literature rejects. Beating it only requires showing a real shift-induced drop. Its error, the gap between source and target accuracy, gives the scale against which every monitor's estimation error is read.

The strongest practical competitor is outcome sampling: wait and label $k$ stream cases with the true diagnosis. Given time, it is unbiased, and it is what prospective validation studies in effect do. We give it exactly the same per-batch budget $k$ as the concept audit. We count cost in two ways: number of annotations, and time until labels become available, with outcomes treated as available only after a follow-up delay. The concept monitor beats it if, at matched annotation count, its accuracy estimate is within a few points of outcome sampling's while being available at reading time. It also beats it if, at matched calendar time, it detects the 10-point drop and outcome sampling cannot yet. It does not beat it at matched annotations with time ignored, and we will report that honestly.

The strongest label-free competitors are mean confidence, entropy, and embedding drift. They are computed on the same models and stream batches, and their accuracy-mapping curves are fitted with the same source-only protocol. They cost no annotation, so the concept monitor must earn its clinician time. On white blood cell, dermatology and shortcut-injected nodule streams, the concept monitor should show clearly lower accuracy-estimation error and fewer missed failures. For these monitors I found no reported result on Lacuna for any of these datasets, so all numbers will be produced here.

On predictive accuracy, the strongest existing reference points come from the file. With a ViT backbone, the clinical alignment loss raised out-of-domain F1 from 43.64% to 49.75% on RaabinWBC, and with a VGG+Linear configuration it raised DDI F1 from 58.87% to 60.13% (Integrating Clinical Knowledge into Concept Bottleneck Models). PN-SAMP reports 97.58% malignancy accuracy and 89.33% attribute accuracy on LIDC-IDRI (Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction). Intermediate-layer distillation reaches AUC 0.733 against ERM's 0.461 on MIMIC (Preventing Shortcut Learning in Medical Image Analysis through Intermediate Layer Knowledge Distillation from Specialist Teachers). Our bottleneck models must land within a few points of these figures, or the monitor is being bought with an unacceptable model.

The ablation that isolates the key ingredient is the black-box model with side findings heads. It has the same backbone, data, findings labels, tuning budget and audit protocol as the bottleneck, but its diagnosis is not forced through the findings. If its findings error predicts its own diagnostic accuracy as well as the bottleneck's does, the bottleneck is unnecessary. A second ablation removes the alignment loss to test whether expert concept ranking matters for monitoring as well as for robustness.

Why not just…?

Why not just monitor confidence or input drift, which needs no clinician? These signals say the inputs look different. They do not say whether the model's answers got worse. A model that has learned a shortcut can stay confidently wrong when the shortcut flips. The distillation paper's ERM model falls to AUC 0.461 on MIMIC, below chance, which is the kind of failure confidence hides. The real threat is that on purely natural shifts such as PBC to RaabinWBC, drift statistics may track accuracy well enough. The head-to-head comparison per shift type will show where clinician audits are actually needed.

Why not just collect a small batch of outcome labels? That is the right final answer when outcomes are quick. In oncology and cardiology the delay is the problem: 530-day follow-up in the OCT study, and 3-year progression-free survival in DSA-Net (Real-time automatic prediction of treatment response to transcatheter arterial chemoembolization in patients with hepatocellular carcinoma using deep learning based on digital subtraction angiography videos). Over that period, a degraded model keeps influencing care. We do not claim concept audits replace outcome validation. We claim they give a same-day estimate whose error relative to eventual outcome sampling can be measured, and the matched-budget comparison measures it.

Why not just add findings heads to the best black-box model, keeping its accuracy and avoiding the distraction effect? This is the most serious alternative, and it is the central ablation rather than a side note. The structural argument says side heads can stay correct while the diagnosis follows a shortcut. ShorT's finding that encoding does not equal usage points the same way. But this is untested. If side heads monitor as well as the bottleneck on the shortcut-injected LIDC streams, we will recommend side heads, and the contribution becomes the audit protocol rather than the architecture.

Why not just make the model robust, for example with distillation or alignment, and skip monitoring? Robustness methods narrow the gap without telling a deployment site whether it has been closed. The best reported out-of-domain numbers still leave large errors, such as 49.75% F1 on RaabinWBC. Monitoring and robustness work together, so we monitor robustified models too.

Decisive Checks

The idea survives only if three results hold. First, on the shortcut-injected LIDC-IDRI streams, the bottleneck's concept monitor must estimate diagnostic accuracy markedly better than the side-head model's concept monitor, at equal audit size. This is the test of the core structural claim. Second, on PBC to RaabinWBC/Scirep and Fitzpatrick 17k to DDI, it must beat the best label-free monitor at a modest audit size, around a few dozen cases per batch, while the bottleneck's accuracy stays near the alignment-loss figures above. Third, on the LIDC stream with outcome-side shift, it must fail as the decomposition predicts. A monitor that also "succeeded" there would mean our calibration is leaking target information.

If the first check fails, the bottleneck is not the active ingredient. If the second fails, clinician audits are not worth their cost. Either result is useful to anyone planning a prospective rollout.