Measuring How Fast a Medical Triage AI Gets Less Safe, With Few Specialist Labels

In clinics with almost no specialists, AI systems sort routine images. A chest X-ray model or a retinal camera model clears the obviously normal cases automatically and sends the rest to a scarce radiologist or ophthalmologist. The safety of such a system rests on one number: how often a sick patient gets cleared by mistake (the miss rate among auto-cleared cases). That number drifts after deployment as patients, prevalence and scanners change. Checking it requires the same scarce specialists to re-read cleared images, so deployments audit tiny random samples, such as the roughly 1% audit behind the ARDA diabetic retinopathy deployment in India. From an audit that small, a health system can say "the miss rate was acceptable over 2019–2023." It cannot say "the miss rate is rising by X% per year, and the rise began when the clinic changed cameras."

We propose an audit-and-estimation procedure that answers the second question. It borrows the epidemiologist's estimated annual percentage change (EAPC): fit a line to the logarithm of a rate over calendar years and report the slope as a percent change per year. It applies this to the auto-clear miss rate. The procedure makes the estimate affordable under a fixed specialist budget by combining two ideas from permutation-free significance testing. The first is a kernel drift statistic computed on unlabeled image embeddings. It decides which cleared cases specialists should re-read, and the sampling probabilities are recorded so the estimate stays unbiased. The second is a cross-fitted, doubly robust correction of the kind used in AIPW-xKTE. It makes the resulting EAPC approximately normal, so a confidence interval and a significance test come from a closed-form formula. The claim is that, for the same number of specialist reads, this yields valid and much narrower intervals on the annual velocity of harm than uniform random audits. It also locates trend breaks sooner.

Core Claim

Prior work treats post-deployment shift in triage AI as a one-off event. Conformal Triage calibrates thresholds once on a local set and is valid only while new data resemble that set. Existing deployment reports, AIRIS-TB on one million chest X-rays and ARDA, give aggregate cross-sectional numbers. Kernel two-sample tests such as cross-MMD and martingale MMD could detect embedding drift cheaply. At deployment scale, however, they will reject for almost any shift, including harmless ones, and they say nothing about patient harm.

We change the object being tested. The question is no longer whether the embedding distribution has changed. It is whether the annual percentage change in the auto-clear miss rate is greater than zero. Unlabeled drift information is spent where it actually helps, which is in choosing which cases the scarce specialists audit.

Research Program

Partition deployment time into windows $t$ (months, aggregated to calendar years for the EAPC). The quantities in each window are defined as follows.

  • $C_t$ is the set of auto-cleared cases in window $t$.
  • $Y_i$ is 1 if cleared case $i$ is in fact abnormal.
  • $R_i$ indicates that a specialist audited case $i$.
  • $\pi_i$ is the known, design-chosen probability of that audit.
  • $g$ is an outcome model that predicts $Y$ from the image embedding. It is fit on a different fold of audited data, which is the sample-splitting device from cross-MMD and AIPW-xKTE that prevents using the same data twice.

The per-window miss-rate estimate is

$$\hat m_t = \frac{1}{|C_t|}\sum_{i \in C_t}\Big[g(x_i) + \frac{R_i}{\pi_i}\big(Y_i - g(x_i)\big)\Big],$$

which is unbiased for any $g$ because the audit probabilities $\pi_i$ are known by design. A good $g$ shrinks the variance. This is the doubly robust structure of AIPW-xKTE, but here the propensity is controlled rather than estimated, so the no-unmeasured-confounding and overlap assumptions that limit that paper hold by construction as long as every $\pi_i$ is bounded below.

The velocity is then the slope $\beta$ of a weighted log-linear fit across years,

$$\ln \hat m_t = \alpha + \beta\, t + \epsilon_t, \qquad \text{EAPC} = 100\,(e^{\beta}-1).$$

Each window's weight is its inverse delta-method variance. The studentized slope is then approximately standard normal under no trend, which gives an analytical test of EAPC $> 0$ with no bootstrap.

Because miss rates are rare (AIRIS-TB reports false negative rates of 0.33% to 1.57%), $\ln \hat m_t$ is unstable when the audited miss count in a window is near zero. We therefore also fit a quasi-Poisson regression on doubly robust pseudo-counts, and we pool windows until each has a minimum expected number of audited events. Joinpoint segmentation is applied to the same pseudo-rates to locate inflections, with segment-weighted average annual percentage change summarizing the whole period.

The targeting rule is where the kernel test does real work. At each window, a cross-MMD witness function is computed between cleared-case embeddings at calibration time and in the current window. The witness function is the direction in embedding space along which the two samples differ most. Audit probabilities are tilted toward cleared cases with high witness values or high $g$, under a floor on $\pi_i$ and a fixed total budget $b$ of specialist reads, echoing the fixed-budget framing of Differentiable Learning Under Triage. The intuition is that harm tends to concentrate in the region the deployment has drifted into, and that is where audit labels reduce variance most.

What we will build and analyze has four parts. The first is the estimator and its asymptotic normality argument, which combines cross-fitting with known propensities. The second is the variance-optimal tilting rule under a budget. The third is an open evaluation harness that simulates years of deployment from public chest X-ray data.

The fourth part is an analysis of when the method fails. We expect it to underperform if the outcome model $g$ is badly misspecified, or if harm arises in regions the witness function does not flag. In those cases the floor on $\pi_i$ guarantees validity but not efficiency.

Proposed Experiments

No public dataset follows a triage model over years with dense labels, so we build semi-synthetic deployment streams with known ground-truth miss-rate trajectories. A triage classifier is trained on CheXpert and thresholded for a high-sensitivity "clear normal" operating point. It is then streamed over MIMIC-CXR, whose 377,110 images from 2011–2016 come with labels extracted from reports (reported F1 of 0.973 for pleural effusion). It is also streamed over the PadChest and TB-11K external sets that the AIRIS-TB study used to span abnormality prevalence from 0.5% to 87.5%. Report-derived labels act as the specialist oracle and are revealed only when a case is audited.

Drift scenarios are injected with controlled timing. The first is a gradual compounding rise in prevalence, which is exactly the log-linear model. The second is an abrupt acquisition change made by switching the source institution mid-stream. It mimics the Siemens-to-GE scanner swap that Schmitz-Koep et al. showed can turn about −0.4% annual change into −2% to −3% depending on the processing pipeline. The third is a benign shift that moves embeddings but leaves the miss rate fixed, created by changing the age mix within normals.

The metrics are four:

  • coverage of the true EAPC by the 95% interval;
  • interval width at a fixed audit budget, with budgets swept from 0.25% to 4% of cleared cases and the ARDA-like 1% as the anchor;
  • detection delay for the injected joinpoint;
  • the false alarm rate for EAPC $> 0$ under the benign shift.

Baselines

The naive reference is ARDA's practice turned into a trend. Specialists audit a uniform random fraction of cleared cases, and a standard EAPC is fit by log-linear regression on the raw audited miss proportions per year. It uses the same total reads, windows and joinpoint software as our method. ARDA reports a 0% clinically important miss rate from 4,537 audited images, but gives no trend estimate. We found no reported result on Lacuna for EAPC estimation of triage miss rates, so this baseline is re-run by us. Beating it means equal or better coverage with a clearly narrower interval at the same budget. We pre-register a target of at least a 30% width reduction at the 1% budget, along with earlier joinpoint detection.

The strongest existing method on the drift side is an unlabeled monitor. Cross-MMD runs on cleared-case embeddings each window, at the analytical cost its authors report as hundreds of times faster than permutation MMD. Its windowed statistic is regressed on time. This monitor is the natural competitor because it costs no specialist time at all. It wins only if its alarms track harm. We beat it if it flags the benign scenario as often as the harmful ones while our miss-rate EAPC does not.

The strongest calibration-side competitor is periodic conformal recalibration in the style of Conformal Triage, which reported cutting false positives from 233 to 8 while abstaining on 14% of cases. It is re-run with the same audit labels used as fresh calibration sets each window, and we report its implied miss-rate trajectory.

Three ablations isolate our key ingredients, all under the same budget and folds:

  • uniform audits with the outcome-model correction, which separates the control variate from targeting;
  • targeting without the correction, using inverse-probability weighting only;
  • targeting by the triage model's own confidence rather than the drift witness function.

Why not just…?

Why not just audit uniformly and run the standard EAPC? This is the real threat when misses are common or budgets are generous, and the budget sweep is designed to show where the crossover lies. At AIRIS-TB-like miss rates of a fraction of a percent, a 1% audit of a site clearing tens of thousands of cases per year yields only a handful of audited misses. Rate estimates built on so few events can swing widely and some windows contain none, which is precisely the regime where we expect variance reduction to matter. If uniform audits match our intervals at realistic budgets, the targeting and the outcome model are not worth their complexity, and we will report that.

Why not just alarm on embedding drift with cross-MMD or martingale MMD? With hundreds of thousands of images, these tests will detect trivial shifts. Their output is a p-value about the inputs, not a rate of patient harm. The benign-shift scenario directly measures this failure. We keep the kernel statistic, but we use its witness function to direct audits rather than treating its p-value as the answer.

Why not just recalibrate conformal thresholds on a schedule? Recalibration restores the guarantee only after a fresh representative labeled set exists, and it consumes the same scarce labels. It also produces a sequence of thresholds rather than a velocity with a significance test, so it cannot tell a manager that savings in specialist time are eroding by a measurable percentage per year. In our comparison, recalibration receives the identical labels. Its role is to show whether our estimator detects deterioration before scheduled recalibration would.

Why not use permutation tests instead of the analytical one? On monthly windows of a few thousand embeddings, permutation is affordable. We will run it as a calibration check on the null distribution. The analytical form matters mainly for repeated cross-fitted slope tests across many windows and sites. The scientific claim does not depend on it.

Decisive Checks

The idea succeeds if three things hold. Coverage of the true EAPC stays near 95% in all four drift scenarios, including when the outcome model is deliberately degraded. Interval width at the 1% budget is substantially below the uniform-audit baseline. The scanner-switch joinpoint is located within fewer windows.

The idea fails if uniform audits do as well, or if the witness-targeted ablation is no better than confidence targeting. Either result would mean the drift test adds nothing beyond the triage model's own uncertainty.