Per-Patient Evidence Appraisal for Non-Invasive Cancer Triage

Blood-based models that could spare people a colonoscopy, gastroscopy or cervical exam often look excellent in the hospital that built them and then lose accuracy elsewhere. A routine-lab colorectal model drops from 0.993 to 0.943 AUROC (area under the ROC curve, a ranking-quality score where 1.0 is perfect) between development and an external hospital. A blood-count cervical model falls from 0.916 to 0.791 accuracy on a prospective external cohort. Systematic reviewers already have a vocabulary for why published studies mislead. Critical-appraisal checklists ask how patients were selected, whether the diagnosis was confirmed by the same test for everyone, and whether exposures were measured comparably. This proposal applies those same questions to each individual prediction at the moment it is made, rather than once per published paper.

The core idea fits in two sentences. When a triage model scores a new patient, it should also report, domain by domain, how well that patient's case is supported by the training evidence. The domains are: is this patient within the range of people whose diagnosis was actually confirmed; were their lab values measured like the training data; and does the training cohort's case-control sampling distort this patient's risk? We hypothesize two things. First, these appraisal signals predict which individual predictions will be wrong under external shift better than generic model confidence does. Second, they give clinicians a concrete, checkable reason to override the model, which post-hoc feature attributions do not.

Core Claim

The invasive gold standard creates the methodological flaw that non-invasive models inherit. A label such as "no colorectal cancer" exists only for people who underwent colonoscopy or biopsy, so training cohorts are built from verified patients. These are often assembled case-control style: the colorectal cohort of Population-based colorectal cancer risk prediction using a SHAP-enhanced LightGBM model has 676 cases against 410 controls, far from screening prevalence. Diagnostic-accuracy appraisal calls these problems spectrum bias and partial verification bias. Standard machine-learning robustness tools cannot detect them, because they concern which labels exist, not which inputs look unusual.

We claim that an appraisal decomposed into three domains, each estimated per patient, will do three things. It will predict cohort-level external performance decay. It will rank individual errors better than confidence or outlier scores. And it will help readers catch wrong advice. The three domains are selection and spectrum, verification, and measurement comparability.

The verification domain is the one generic out-of-distribution detection cannot replicate. If it adds nothing beyond covariate-distance scores, the proposal reduces to relabeled outlier detection, and we will report that.

Relation to Prior Work

The motivation comes from three places. Artificial Intelligence Performance in Image-Based Cancer Identification: Umbrella Review of Systematic Reviews appraised 158 reviews of diagnostic AI. It found that 80.4% met high methodological standards, yet only 5% of the underlying evidence reached high GRADE certainty. It names retrospective design and opacity as translation barriers but builds no mechanism.

Appraisal tools such as the Newcastle-Ottawa Scale are applied per study to filter meta-analyses, as in Seizure outcome-related factors in autoimmune encephalitis and Gait Parameters and Peripheral Neuropathy in Patients With Diabetes. We keep their domain structure but move it from the study level to the prediction level. We also make it estimable from data rather than scored by a reviewer.

DEPLOYR documents retrospective-to-prospective decay in live silent trials, for example a white-blood-cell model falling from 0.76 to 0.69 AUROC. It monitors aggregate drift on a dashboard but does not explain individual predictions. We aim to anticipate that decay per patient.

On the interpretability side, the colorectal and cervical models use SHAP, a method that attributes a prediction to its input features. SHAP says which features drove the score. It does not say whether the score deserves trust, and it cannot flag that the training controls were a non-representative verified subset.

Integrating Clinical Knowledge into Concept Bottleneck Models shows that constraining reasoning toward clinical knowledge improves out-of-domain F1 (43.64% to 49.75% on RaabinWBC). That supports the premise that structure tied to clinical reasoning aids robustness. We deliberately do not constrain the predictor itself: the distraction effect reported by Integrating Clinically-Relevant Features into Skin Lesion Classification warns that forcing concepts into the model can cost accuracy. Our appraisal layer sits beside an unchanged predictor, so it cannot degrade discrimination.

Finally, Do as AI say found that 27.54% of radiologists and 41.73% of non-experts followed incorrect advice every time, while rating AI advice lower. This constrains design: an explanation that merely makes a prediction feel justified could worsen this. Our reader study therefore measures override of wrong advice, not trust.

Research Program

We build three estimators around any fixed triage model.

The selection and spectrum estimator has two parts. It fits a density-ratio model between the verified training population and the deployment population's feature distribution. It also corrects the predicted risk from training case-control prevalence to target prevalence using the standard prior-shift adjustment. It reports both the size of the correction and the patient's position relative to the verified spectrum.

The verification estimator is the key ingredient. It uses records that are available in any EHR but absent from published cohorts: patients with the same routine labs who never received the invasive test. From these it fits $\pi(x)$, the probability that a patient with features $x$ would have been verified by the gold standard. Low $\pi(x)$ means the model's behavior for this patient rests on few or selectively confirmed labels. Inverse-$\pi$ weighting also gives a verification-corrected risk estimate. The disagreement between that estimate and the model's raw score is itself a per-patient signal.

The measurement estimator flags individual lab values whose analyzer, unit or distribution differs from training. It works per feature, drawing on laboratory metadata where available, so the report can state "hemoglobin measured on a platform not seen in training" rather than an anonymous outlier score.

The output is a short structured report attached to each prediction, with one graded judgement per domain in the style of an appraisal checklist, plus a combined certainty grade. We will fit the combination rule on development-site data only. We will then test whether it forecasts the error rate at held-out sites, which is the property GRADE-style certainty is supposed to have and, per the umbrella review, rarely demonstrates.

Baselines

The naive reference point is to take the model's internal validation performance at face value and treat every prediction as equally trustworthy. On the colorectal task the published figures are 0.9931 AUROC in training and 0.9429 externally (Du et al.). On the cervical task they are 0.916 internal and 0.791 external accuracy with 0.740 sensitivity (cervical CBC model). Any appraisal signal must beat this reference by identifying where those losses concentrate.

The strongest existing competitor for per-prediction trust is conformal prediction. It wraps any model and returns prediction sets with a coverage guarantee under exchangeability, and it is implemented in PyHealth 2.0, which reports about 99% empirical coverage on its test sets. We found no reported result on Lacuna for conformal or confidence-based selective prediction on these cancer triage cohorts. We will run it with the same predictor, the same calibration split and the same development-site data. Beating it means a higher AUROC for predicting per-patient error at external sites, and a better risk-coverage curve (error rate among the predictions retained as more uncertain ones are deferred).

The second strong competitor is generic outlier detection: Mahalanobis distance and a density-ratio classifier on raw features, tuned with the same budget as our selection estimator. No reported result was found on Lacuna for this on these cohorts.

The explanation baseline for the reader study is TreeSHAP as deployed by Du et al.

The decisive ablation removes the verification estimator and keeps everything else. If that removal does not hurt error prediction on external cohorts and in the controlled simulation, the key ingredient has failed. A second ablation collapses the three domains into one scalar score. That tests whether the decomposition matters for readers even if it does not matter for error ranking.

Why not just…?

Why not just use conformal prediction or model confidence? Conformal guarantees depend on calibration data resembling deployment data. Partial verification violates this in a way calibration cannot see: the calibration set is itself drawn from verified patients. Confidence is a real threat on covariate-shift errors, and we expect it to be competitive there. The simulation below separates the cases. If conformal matches us even when labels are biased and covariates are not shifted, we concede the point.

Why not just run an outlier detector? This is the most serious alternative, because our selection and measurement estimators are partly outlier detectors with names attached. What it cannot capture is verification bias with matched covariates. Examples are controls who were verified because a clinician suspected disease for reasons absent from the lab panel, or case-control prevalence distortion that leaves the feature distribution unchanged. The verification ablation and the matched-covariate simulation settle whether this extra component earns its place.

Why not just recalibrate to the target prevalence? Prior-shift recalibration fixes average miscalibration from case-control sampling and is included inside our selection estimator. It does not change the ranking of patients, so it cannot explain AUROC decay like the cervical model's external sensitivity drop. It also says nothing about which patients are poorly supported. We will report recalibration alone as a baseline for calibration and net-benefit metrics.

Why not just build an intrinsically interpretable model, such as sparse logistic regression on the 13 labs? For tabular panels this may match gradient boosting, and we will check it. However, a transparent model trained on a spectrum-biased, partially verified cohort is transparently wrong in the same places. Interpretability of the mechanism does not address the provenance of the labels, which is our target.

Proposed Experiments

The first experiment is a controlled testbed with known ground truth. From MIMIC-IV via PyHealth 2.0, we define a lab-based diagnosis task. We then deliberately construct development cohorts with case-control sampling, feature-dependent verification, and verification that depends on an unrecorded variable, while holding out an unbiased target cohort. This is the only setting where we can show the decisive result: under verification bias with matched covariates, our verification estimator predicts error while conformal and outlier baselines do not. The kill criterion is simple. If the verification estimator does not improve error-prediction AUROC over outlier detection in the unrecorded-variable condition, we drop it.

The second experiment uses real external shifts: the colorectal Guigang-to-Laibin pair, the cervical internal-to-prospective pair, and, where access permits, the gastric cell-free DNA cohorts of Yu et al.. We retrain each published model family on development data. We then measure three things: correlation between the combined certainty grade and observed per-stratum external error; error-prediction AUROC against conformal and outlier baselines; and decision-curve net benefit for referral to the invasive test at realistic screening prevalence. The prediction we can fail is that the appraisal grade forecasts external decay, including a drop from 0.916 to about 0.79 accuracy on the cervical task, from development data alone.

The third experiment runs the appraisal layer silently on STARR laboratory data in the DEPLOYR style, where unverified patients are abundant. It tests whether predictions flagged as poorly verified account for a disproportionate share of the retrospective-to-prospective AUROC gap.

The fourth experiment is a reader study that replicates the factorial design of Do as AI say with triage vignettes: mostly correct advice, some realistic errors, and three arms (score only, score with SHAP, score with appraisal report). The primary outcome is the rate of following incorrect advice, benchmarked against the 27.54% and 41.73% highly-susceptible rates reported there. Adherence to correct advice is tracked so that improved override rates do not come from blanket distrust.