Spending Chart Review Where the Treatment Effect Is Uncertain

Retrospective cohort studies estimate whether a treatment helps by mining historical health records. Their outcomes, though, usually come from billing codes or text classifiers that are often wrong, and the only reliable fix is a clinician reading the chart, which is slow and expensive. Today the charts that get reviewed are almost always picked at random or by a fixed rule. This note proposes choosing them deliberately. Review should go to the patients whose true outcome would most change the treatment-effect estimate. These are patients whose automated label is ambiguous and who also carry large weight in the causal estimate because, given their profile, they rarely receive the treatment they actually got.

The technical device is adaptive stratified sampling: spend a fixed budget across subgroups so as to shrink the error of a global estimate, while learning on the fly which subgroups are noisy. Existing adaptive designs target the wrong quantity for this setting. They minimize error in an outcome average, as PartiBandits does on a 6-million-patient record database, or they assume the analyst controls who gets treated, as OPTrack and ClipSMT do. We instead derive the review allocation from the variance of the causal estimator itself. We then run it in clinician-sized batches with confidence intervals that stay valid.

Core Claim

In a retrospective cohort, the treatment was assigned long ago, so the only design choice left is which charts to verify. Suppose every patient $i$ has covariates $X_i$, an observed treatment $T_i$, a cheap surrogate outcome $S_i$ (a billing code or a classifier score), and a true outcome $Y_i$ revealed only by review. A natural estimator does three things. It imputes outcomes for everyone from a model $m(X,T,S)$ fit on reviewed charts. It forms the standard doubly robust treatment-effect estimate, which combines an outcome model with inverse-propensity weighting so that it stays consistent if either part is correct. Finally, it adds back the residual $Y_i - m(X_i,T_i,S_i)$ on reviewed charts, divided by each chart's review probability $\pi_i$.

Write $e(X)$ for the propensity score, the probability of treatment given covariates, and $w_i = T_i/e(X_i) - (1-T_i)/(1-e(X_i))$ for patient $i$'s inverse-propensity weight. The extra variance contributed by not reviewing every chart is then proportional to $\sum_i w_i^2 v_i / \pi_i$. Here $v_i$ is the conditional variance of the true outcome given the surrogate and covariates. Minimizing this sum subject to a budget of $n$ reviews, using a Lagrange multiplier exactly as in classical Neyman allocation, gives

$$ \pi_i \;\propto\; |w_i|\,\sqrt{v_i}. $$

This rule says something that neither literature currently does. Pure outcome-variance allocation, as in PartiBandits, sets $\pi_i \propto \sqrt{v_i}$ and ignores the weight. It therefore under-reviews exactly the poor-overlap patients who dominate the variance of the causal estimate, such as the elderly comorbid men who rarely got surgery in Tewari et al.. Pure propensity stratification ignores $v_i$ and so wastes reviews on patients whose code is already decisive.

Our claim is twofold. First, targeting $|w_i|\sqrt{v_i}$ yields materially lower error in the treatment effect per review than either rule alone. Second, the gain grows as propensity overlap worsens. Because $v_i$ is unknown and must be learned from the reviews themselves, the second ingredient is how to learn it. We use upper-confidence estimates, motivated by OPTrack's result that under-sampling costs far more than over-sampling. We also floor each $\pi_i$ away from zero, as ClipSMT does for treatment probabilities.

Research Program

The allocation is built over strata defined by crossing three things: propensity-score bins, surrogate-score bins, and treatment arm. Within each stratum, $v_i$ is estimated from the reviewed charts' residuals. Review proceeds in batches of 50 to 100 charts, matching how clinicians actually adjudicate. A small uniform pilot batch comes first. Each later batch is allocated by the rule above, plugging in an upper confidence bound on each stratum's residual standard deviation, with a minimum review probability in every stratum.

Because each batch's probabilities depend only on previously reviewed charts, the weighted residual corrections form a martingale difference sequence. The adaptive-weighting argument behind the A2IPW estimator used in OPTrack then supplies a central limit theorem and valid Wald intervals. We will prove this for the batched, stratified retrospective case and state the rate under which the plug-in allocation approaches the oracle.

The surrogate model $m$ can be as simple as a stratum-level misclassification correction. In the binary-outcome case, that correction is the Rogan-Gladen inversion that Lee et al. use to calibrate LLM judges from a small human set. Our design reuses their calibration idea. The difference is in what decides the human set: Lee et al. choose it by which proxy class shrinks a win-rate interval fastest, whereas here it is chosen by influence on a causal contrast.

The scope is deliberately narrow. The outcome is misclassified; treatment and confounders are taken as recorded. The cohort is binary-treatment. Misclassified exposures are a stated extension, not a promise.

Baselines

The naive reference point is the surrogate-only analysis. It runs the same doubly robust estimator with the billing code treated as the true outcome and no review at all, which is how many retrospective studies proceed in practice. Any review design must beat it on error. The interesting question is by how much at small budgets.

The first real competitor is simple random review plus the same augmented estimator. This baseline isolates allocation, because the estimator, cohort, budget and outcome model are identical and only which charts get read differs. Beating it means a lower root-mean-squared error against the full-review target at equal budget, or equivalently reaching that baseline's error with fewer reviews. We count a reduction of at least 25% in required reviews as meaningful, since Kumar et al. report savings up to 60% for variance-guided allocation in model evaluation. A transfer of Lee et al.'s Rogan-Gladen calibration, using a random calibration set, is run as a variant of this baseline.

The strongest existing adaptive method is PartiBandits with its WarmStart-UCB subroutine (Neopane et al.), reimplemented to allocate reviews by outcome variance across learned strata. It receives the same covariates and surrogates, the same budget, and the same batch structure, with its partition-learning half retained. Its reported AFC result is only qualitative: lower error than random sampling across budgets. For causal-effect estimation from chart review on our benchmark, no reported result was found on Lacuna. We therefore re-run it ourselves under matched conditions.

The second adaptive competitor is uncertainty-driven chart selection in the spirit of RELEAP (Yang et al.). It reports logistic-regression AUC rising from 0.774 to 0.805 on the Duke cohort, but that is a prediction target, not a causal one. We implement only its uncertainty-sampling heuristic rather than the full PPO agent, and we note this simplification.

Three ablations isolate the key ingredients. The first drops the $|w_i|$ factor, which reduces our method to outcome Neyman allocation. The second drops $\sqrt{v_i}$, which reduces it to propensity-weighted allocation. The third replaces optimistic batch updates with a single two-stage plug-in allocation computed from the pilot.

Why not just…?

Why not just review a random sample and let the doubly robust correction fix the bias? This alternative gives unbiased answers, and our method keeps exactly that correction. Our allocation buys only variance. The random-review baseline above measures that gain directly. If the gain is small in well-overlapped cohorts, we will say so. The theory predicts the advantage concentrates where $|w_i|$ is heavy-tailed, and the overlap sweep below tests exactly that.

Why not just stratify once using a pilot and then fix the allocation? This is the real threat. A two-stage design with the same $|w_i|\sqrt{v_i}$ target may capture most of the benefit. ClipSMT finds explore-then-commit degrades when the optimal allocation is far from uniform, but that finding comes from prospective treatment assignment, not chart review. The two-stage ablation settles the question. If it comes within about 10% of the batched optimistic version, the deliverable becomes the two-stage design with the causal target. That is still new relative to PartiBandits and Lee et al., because the novelty lies in what is being minimized rather than in the bandit machinery.

Why not skip review and model the surrogates semiparametrically? TUBE reaches AUC 0.894 for phenotyping without large gold-standard sets. Even so, LATTE still needed 100 gold labels to outperform its next-best method by 11.0%. Residual misclassification that is correlated with treatment biases the effect, and a model of surrogates alone cannot detect that. The surrogate-only arm quantifies this bias on real data.

Proposed Experiments

The primary testbed is MIMIC-IV-Ext-PE. It contains 19,942 CT pulmonary angiography reports, every one with a gold pulmonary-embolism label. It also carries two realistic surrogates with reported accuracy: billing codes at 95.4% sensitivity and 83.8% positive predictive value, and VTE-BERT at 92.4% and 87.8%. Billing codes additionally miss some 7,500 emergency department records. Because every report has a gold label, chart review can be simulated exactly, and the full-review effect estimate serves as ground truth.

We will pre-specify, before inspecting outcomes, an exposure from structured MIMIC-IV fields, for example a recorded anticoagulant order before the scan. We will also pre-specify a confounder set. We then estimate that exposure's association with confirmed embolism, treating it as a methodological target rather than a clinical claim. Budgets range from 200 to 2,000 reviews, with 500 repeated designs per method. The metrics are root-mean-squared error against the full-review estimate, 95% interval coverage, interval width, and reviews needed to match random review's error at 1,000 charts.

The decisive check is a semi-synthetic overlap sweep. We keep the real covariates, surrogates and gold outcomes, but resample treatment from propensity models whose tails we control. The method succeeds if its advantage over both outcome-only Neyman allocation and random review grows monotonically as overlap shrinks, while coverage stays near 95%.

A secondary simulation, calibrated to the reported surrogate error rates, checks the theory's rate predictions and batch-size sensitivity at batches of 1, 50 and 200. A sanity replication of PartiBandits' prevalence-estimation setting confirms our reimplementation behaves as published before it is used as a competitor.

Two outcomes would reject the core claim. The first is that dropping the $|w_i|$ factor costs nothing under poor overlap. The second is that coverage fails under batched adaptivity.