Tired Doctors as a Natural Experiment for Missing Labels

Clinical prediction models are usually trained on hospital records, and in those records a patient has a diagnostic label only if a clinician chose to order the test. A model that predicts "heart attack" from emergency department data has therefore learned about the patients doctors decided to test, not about all patients. How doctors make that decision depends on their state of mind as well as on the patient. Clinical Notes Reveal Physician Fatigue found that the yield of acute cardiac testing falls by 18% for each standard deviation of predicted physician fatigue. Tired doctors test more liberally and catch fewer positives. That finding is usually read as a problem: fatigue contaminates the data. This proposal reads it as an opportunity as well. If the physician's workload moves the testing threshold up and down while the patients stay comparable, then busy stretches reveal outcomes for lower-risk patients who would normally go untested. Comparing test yields across workload levels can then estimate, or at least bound, the disease rate among the patients who were never tested.

In two sentences: clinician workload changes who gets tested, which biases any model trained on test results. The same variation, if it comes from the schedule rather than the patients, works like a natural experiment that lets us correct the resulting missing-label bias beyond what standard reweighting on patient features can do.

Core Claim

The retrospective selection-bias literature treats the process that generates records as a matter of institutional policy, survival, and demographics. To weight or not to weight? reweights three large biobanks so that they match census and survey benchmarks. It shows that this corrects disease prevalence (median prevalence ratio 0.61 in the Michigan Genomics Initiative) but is blind to selection on unmeasured variables. The fatigue study, for its part, treats fatigue as an outcome and as an explanation for yield and racial disparities. It does not ask what fatigue does to models trained on the resulting labels. Our claim is narrower and testable. First, testing decisions in retrospective clinical data depend on clinician workload in a way that patient features cannot absorb. Second, a model trained only on tested patients therefore inherits a miscalibration that shifts with workload. Third, workload that is fixed in advance by the schedule, such as the number of shifts the physician worked in the preceding seven days, can serve as an instrument: a variable that changes the testing rate but has no direct effect on disease. This instrument yields bounds on risk among untested patients that are tighter and more accurate than those from propensity reweighting on patient features.

The depletion literature constrains how we state the mechanism. The pre-registered null in No Evidence of the Ego-Depletion Effect (d = 0.22, Bayes factor 2.40 favoring the null) rules out building anything on the "draining reservoir" model. We borrow only the weaker, better-supported view from Neural and Behavioral Evidence for an Intrinsic Cost of Self-Control: effortful deliberation has a subjective cost, so decision rules shift when that cost rises. Regulation under conflict itself consumes bandwidth, and Regulating Emotions during Difficult Multiattribute Decision Making showed that depletion impaired coherence shifting in the lab. That makes it plausible that a strained clinician resolves diagnostic conflict by ordering the test rather than reasoning past it. Nothing in the method depends on this story, however. We measure the threshold shift directly and treat it as the empirical first stage, meaning the step that estimates how strongly workload moves the testing rate.

Research Program

The estimator rests on one accounting identity. Let $x$ be the patient's recorded features at arrival, $z$ the physician's pre-shift workload, $p(x,z)$ the probability of being tested, $m(x,z)$ the positive rate among tested patients, and $u(x,z)$ the unobserved positive rate among untested patients. If workload does not change the patient's true disease prevalence $\mu(x)$, then

$$\mu(x) = p(x,z)\,m(x,z) + \big(1-p(x,z)\big)\,u(x,z) \quad \text{for every } z.$$

When workload raises testing from $p_0$ to $p_1$ at fixed $x$, the patients added at the margin have risk

$$r(x) = \frac{p_1 m_1 - p_0 m_0}{p_1 - p_0}.$$

This is the logic of marginal-treatment-effect estimation applied to test ordering. Suppose clinicians rank patients by perceived risk, which is a monotone-threshold assumption we test below. Then untested patients have risk no higher than $r(x)$, so $u(x,z_1) \le r(x)$ is an upper bound on untested risk. Two observations bound it from below: the untested cannot have negative risk, and prevalence must be the same at every workload level. Together these give an interval for $\mu(x)$. We fit $p$ and $m$ as flexible models with physician and hour-of-week fixed effects. We pool the marginal-risk estimates across physicians within strata of $x$. We then retrain the risk model on reweighted, bound-imputed targets so that it predicts $\mu(x)$ rather than $m(x)$.

The work has three parts. The first is a semi-synthetic benchmark in which we know the truth. It is built on the mortality task of Yet Another ICU Benchmark, where every stay has an observed label. On top of this task we impose simulated "reveal only if tested" rules. In these rules, simulated clinicians with scheduled workloads apply thresholds that shift with workload, optionally differ by patient subgroup, and depend partly on unrecorded severity. The second part is an empirical study on an emergency department cohort of the kind used by Hsu et al. (129,228 visits, 60 attendings, 11,592 shifts). That cohort is not public, so the study requires collaboration. There we estimate the first stage, fit the correction, and validate it against outcomes that are recorded whether or not the patient was tested, wherever record linkage allows. The third part is a subgroup analysis. It asks whether the "mean discrepancy" that Investigating Data Interventions for Subgroup Fairness identified as the bottleneck for subgroup fairness is partly a selection artifact. Hsu et al. report fatigue scores 12% and 21% higher for notes on Black and Hispanic patients, which would mean these groups are selected into testing under different thresholds.

The main identification risk is that busy periods bring different patients. Same-shift census does, so we deliberately avoid it as the instrument and use only schedule-determined workload from before the shift. As an ablation we also run same-shift workload, on the expectation that it will fail the balance checks.

Baselines

The naive reference trains the risk model on tested patients only and treats their positive rate as the risk. This is what most retrospective nomograms do, including the imaging and surgical nomograms in the file. It sets the floor that every correction must beat.

The strongest existing competitor is inverse-probability weighting for test ordering. We model the probability of testing from patient features and weight tested patients by its inverse. This is the within-cohort analogue of the weighting in To weight or not to weight? and is the standard missing-at-random correction. A stronger variant adds shift time, prior-week workload, and the note-derived fatigue score from Hsu et al. as covariates in the testing model. For fairness, both variants use the same feature set, the same model class (logistic regression and LightGBM, as in Tan et al.), the same number of hyperparameter trials, and the same train and test splits as our method. The third baseline is post-hoc isotonic calibration within subgroups, which Tan et al. found worked best against subgroup mean discrepancy. The reference ceiling is an oracle trained on the full labels. In the benchmark, the YAIB paper reports mortality AUROC between 0.84 and 0.87 across datasets for its baseline models. For the ED cohort, the only reported number is the fatigue classifier's AUC-ROC of 60.1%, and no reported result was found on Lacuna for any selective-label correction on this cohort or on YAIB.

The key ablation replaces the workload instrument with a placebo. We shuffle physician schedules across weeks, which leaves the model structure intact but removes the real threshold variation. If the placebo matches the real instrument, then the gains come from extra modeling flexibility, not from workload-driven identification.

Because discrimination can be fine while calibration is wrong, we do not judge success mainly by AUROC. We count the method as beating inverse-probability weighting if it meets three conditions. On the semi-synthetic benchmark it must cut the gap between predicted and true risk among untested patients, measured as calibration error on the full-label test set, with confidence intervals that exclude zero, when selection depends on unrecorded severity. It must not be worse when selection is ignorable. On the ED cohort it must predict linked follow-up outcomes among untested patients better than both weighting variants.

Why not just…?

Why not just reweight by the probability of being tested? When testing depends only on recorded features, reweighting is consistent, and our semi-synthetic "ignorable" condition should show it matching our method. The threat is real in that regime. It breaks down when clinicians test on impressions that never reach structured data, which is exactly the unmeasured-selection limit that Salvatore et al. concede. Adding workload as a covariate in the testing model does not fix this. It makes the testing model more accurate but still assumes that untested patients resemble tested ones with the same features. Our method instead uses workload to learn how the untested differ. The benchmark conditions with unrecorded severity settle which method is right.

Why not just add shift timing or the note-based fatigue score as a feature in the risk model? Doing this changes what the model conditions on, not who carries a label, so it cannot recover risk for untested patients. Worse, the note-based fatigue score is measured after the encounter and partly reflects the patient's presentation. That makes it a bad control and a bad instrument. Hsu et al. also report that shift timing alone captures fatigue poorly relative to note-based measures. We still include both features as a baseline so the data can speak.

Why not just use outcomes that don't depend on testing, such as 30-day death or return visits? Where such linkage exists, it is the better ground truth, and we use it to validate. It is not a replacement for our method for three reasons. It measures a different target from "disease present at the visit." It is available for only a subset of patients. And most retrospective registries, including the SEER-based and imaging cohorts in the file, have no such linkage. If our bounds disagree with linked outcomes, the method fails.

Why not just pool more hospitals? Tan et al. show that naive data addition often fails to improve, or harms, subgroup performance because of distribution shift. More data also leaves the selection mechanism intact, and it narrows confidence intervals around a biased estimate, which is the big-data paradox Salvatore et al. warn about.

Proposed Experiments

In the semi-synthetic benchmark, run on MIMIC-IV and eICU through YAIB, we vary three factors: how strongly workload shifts thresholds, whether thresholds differ by subgroup, and how much selection depends on unrecorded severity. The result is a grid on which each method's calibration error among untested patients, AUROC against full labels, and subgroup mean discrepancy can be compared to the oracle.

The first decisive check is a dose-response curve. The advantage of our method over inverse-probability weighting should grow with the unrecorded-severity share and vanish when selection is ignorable. If it never grows, the idea offers nothing beyond reweighting.

On the ED cohort, the first test is the first stage. It asks whether prior-week scheduled workload moves testing rates within physician, holding hour-of-week fixed. Two checks follow. The balance check asks whether the same workload predicts patient age, acuity, and chief complaint; if it does, the instrument is invalid. The monotonicity check asks whether yield falls smoothly as testing rises.

We then compare corrected models against linked outcomes for untested patients. Finally, we test whether the corrected models shrink the Black-White and Hispanic-White calibration gaps more than isotonic calibration does. That result would show that part of what Tan et al. attribute to distribution shift is in fact selection that depends on clinician workload.

A null first stage would itself be informative. It would mean the 18% yield effect runs through same-shift patient mix rather than through clinician state, which would undercut the fatigue interpretation of Hsu et al.