Retrospective cohort studies estimate treatment effects from records that routine care has already produced. They almost always treat patients as independent draws, with each patient's treatment explained only by that patient's own characteristics. That assumption breaks whenever an institution learns as it goes. In the first COVID-19 wave, for example, a hospital that saw early-intubated patients die may have shifted toward waiting, and then shifted back. When that happens, the chance that patient 400 is treated depends on what happened to patients 1 through 399. Statistically, the cohort is then the log of an unrecorded bandit algorithm. A bandit is an allocation rule that steers future choices toward options that looked good on past outcomes. Work on adaptively collected data shows that this kind of feedback makes standard estimators non-normal and makes their confidence intervals cover too rarely, even when the allocation probabilities are known exactly.
We propose to treat a retrospective cohort, ordered by admission date, as an adaptive experiment with an unknown policy. The program has two parts. The first is a test for whether a cohort shows learning across patients. The second is an estimator that stays valid when it does. The estimator's key move is to fit the treatment-probability and outcome models for each patient using only patients admitted earlier. This forward-only fitting preserves the martingale structure that adaptive inference depends on, which standard cross-fitting destroys. Here "martingale" means a sequence whose next increment has mean zero given the past; the central limit theorems that adaptive inference uses require it.
Core Claim
The dependence that matters here is between patients, not within one patient's trajectory. Epidemiology already has tools for a clinician reacting to one patient's deteriorating vital signs. It has no standard tools for an institution reacting to the outcomes of its previous patients. Our claim is that cross-patient learning is common in exactly the settings where retrospective studies are most valued: new diseases, new drugs, and treatments that are unethical to randomize. Under such learning, propensity-score analyses with sandwich standard errors can undercover. This holds even when calendar time is adjusted for, because allocation drifts toward near-deterministic treatment in some patient groups.
We also claim that a doubly robust estimator repairs coverage when it is built in two specific ways. It must use nuisance models fit forward in time. It must also weight each patient by an estimated conditional standard deviation, the stabilizing device of contextual-bandit inference. The cost is wider but honest intervals. If adjusting for calendar era alone restores nominal coverage across realistic learning strengths, the inferential claim is false and the contribution shrinks to the diagnostic test. The experiments are designed to tell these outcomes apart.
Grounding in Prior Work
The adaptive-inference literature supplies the method, but it assumes the logging probabilities are recorded. Post-Contextual-Bandit Inference builds the contextual adaptive doubly robust estimator (CADR). CADR weights each observation by an estimated inverse conditional standard deviation and recovers normality through a martingale central limit theorem. It needs known propensities and a minimum exploration rate. Statistical Inference with M-Estimators on Adaptively Collected Data extends square-root importance weighting to regression-type estimators, again with known allocation probabilities. Policy Learning with Adaptively Collected Data gives weight schedules tied to a known exploration floor. It also proves that learning a good policy becomes infeasible in the worst case once exploration decays faster than $1/t$. That result constrains us: in patient groups where clinicians converge to always or never treating, no estimator can recover the full-population effect.
Semiparametric Efficient Inference in Adaptive Experiments supplies the propensity truncation schedule we reuse. It also supplies anytime-valid confidence sequences, which we treat as an extension. Demystifying Inference After Adaptive Experiments warns that forcing normality through reweighting gives up power. We accept that cost deliberately, because clinical reporting expects intervals on a familiar scale. All of these papers differ from our setting in one decisive way: their logging policy is an algorithm with recorded probabilities. Ours is a human institution whose probabilities must be estimated.
The cohort literature motivates the problem. Timing to intubation COVID-19 patients: can we put it off until tomorrow? studies 592 ventilated patients from the first wave and reports an adjusted hazard ratio of 1.018 per day of delay (95% CI 1.005–1.03) from a static Cox model. The authors themselves note that protocols shifted during the period. Long-Term Survival in Men With High Grade Prostate Cancer uses baseline propensity matching over a 17-year window in which treatment habits plausibly evolved.
The closest bridge is Towards Regulatory-Confirmed Adaptive Clinical Trials. It designs prospective trials that randomize first and then allocate actively, and it uses retrospective cohorts (PharmGKB, SIVEP-Gripe) only as simulation substrates. We go the other way. We perform inference on cohorts whose adaptation was never designed and never randomized.
Research Program
The first component is a test for cross-patient adaptivity. Patients are ordered by admission. We fit a treatment model that includes the patient's own covariates, a flexible function of calendar time, and summaries of recent outcomes at the institution, such as the observed mortality among the last $m$ treated and untreated patients. Adaptivity is declared when the outcome summaries predict treatment beyond calendar time and covariates. Significance comes from a permutation scheme that shuffles outcomes within calendar blocks. This test is useful even without the estimator, because it tells an analyst whether the independence assumption deserves scrutiny.
There is a confounding risk here. Recent mortality also proxies for hospital strain, which may affect outcomes directly. We therefore include occupancy measures where the data has them. We also interpret the outcome summaries as variables the propensity model must contain, not as a causal story about clinician psychology.
The second component is the estimator. For patient $i$, the treatment-probability model $\hat e_i$ and the outcome model $\hat\mu_i$ are fit only on patients $1,\dots,i-1$. These estimates are therefore fixed given the past, which keeps each patient's doubly robust score centered given history. CADR's stabilizing weights are then applied. Propensities are truncated using the schedule of Cook et al. Whenever truncation binds, we report the overlap-restricted estimand explicitly, because the feasibility bound of Zhan et al. means we cannot honestly claim more.
The endpoint is a fixed-horizon binary outcome, such as 28-day mortality, which is almost fully observed in the intubation setting. Right-censored survival is deferred to a later stage.
The third component is theory. We will prove asymptotic normality under conditions that keep the cumulative error from forward-fit propensities small relative to $\sqrt{n}$. This is the analogue of known-propensity results, with an added rate condition on how fast the online propensity model converges.
Baselines
The naive reference is the analysis cohort papers actually run. That means a propensity-score or regression adjustment for baseline covariates, a doubly robust estimate, and a sandwich standard error treating patients as independent. It is fit on identical data and covariates, so it shows how much the field's default undercovers.
The strongest practical competitor is the same doubly robust estimator with flexible calendar-time adjustment and cross-fitted nuisance models, tuned with the same learner family and hyperparameter budget as ours. It is the strongest competitor because it is what a careful epidemiologist would do about drift, and it may already absorb most of the institutional learning. Beating it means achieving nominal 95% coverage in simulated regimes where it falls short, without more than a modest increase in interval width. Bias must stay comparable.
The strongest method from the adaptive literature is CADR given the true simulated propensities. It is an oracle that real data never permits. On OpenML-CC18 bandit tasks, Post-Contextual-Bandit Inference reports near-95% coverage for CADR, against roughly 70% for standard IPW and doubly robust estimators. We aim to approach that oracle's coverage with estimated propensities.
We also include simulation-based inversion in the style of Simulation-Based Inference for Adaptive Experiments, applied to the estimated policy. It may be more powerful where its model of the policy holds.
Two ablations isolate the key ingredients. The first swaps forward-only fitting for standard cross-fitting while keeping the stabilizing weights. The second keeps forward fitting but removes the weights.
Why not just…?
Why not just adjust for calendar time and use robust standard errors? This is the real threat. If institutional learning acts mainly as a smooth shift in treatment rates, calendar adjustment fixes the point estimate. The open question is the variance when allocation becomes nearly deterministic for some patient groups, since that is where the bandit literature finds non-normality even with known propensities. Our simulations sweep learning strength and the speed at which exploration vanishes. If the calendar-adjusted baseline covers nominally across that sweep, we will report it and drop the estimator claim.
Why not just use marginal structural models or sequential target-trial emulation? Those methods handle a clinician reacting to one patient's own evolving state. They still assume patients are independent of each other. Our dependence runs between patients, so the two problems are complementary. Where data allow, the estimator can be layered on a within-patient time-varying analysis. The ablations show whether the between-patient correction matters once within-patient confounding is handled.
Why not just bootstrap patients? Resampling patients assumes they are exchangeable, which is the assumption under question. A bootstrap that keeps the time order would need to re-simulate the institution's policy, and that is the simulation-based baseline we already include. We will compare its coverage against ours directly.
Why not just plug estimated propensities into CADR with ordinary cross-fitting? Cross-fitting uses future patients to estimate earlier patients' propensities. Under adaptivity, future patients carry information about earlier outcomes, so each score is no longer centered given the past. The cross-fitting ablation tests exactly this. If cross-fitting performs as well as forward fitting, the main technical ingredient is unnecessary, and we will say so.
Proposed Experiments
The first experiment is semi-synthetic, built on real covariates. We use the patient covariates of the SIVEP-Gripe registry (over 11,000 hospitalized COVID-19 patients, as used by RFAN) and of the PharmGKB warfarin cohort. Simulated outcomes have known effects. Simulated institutions adapt in three ways: like a Thompson-sampling bandit, like a clinician heuristic that shifts toward a treatment after a run of deaths, and with drift that does not depend on outcomes, which serves as a negative control for the test. Metrics are coverage of nominal 95% intervals, interval width, bias, and the size and power of the adaptivity test. No reported result on Lacuna exists for coverage on these cohorts, so all comparisons are internal.
The second experiment is the established bandit benchmark. We use the OpenML-CC18 contextual-bandit conversion from CADR, hide the logging probabilities from all estimated-propensity methods, and compare against CADR's reported near-95% coverage under known propensities.
The third experiment is a real-data diagnostic. We run the adaptivity test on SIVEP-Gripe treatment decisions such as ICU admission and invasive ventilation, ordered by admission within region. If access to the Hospital das Clínicas cohort can be obtained, we re-estimate the per-day intubation-delay effect and ask whether its reported interval of 1.005–1.03 survives forward-fit, stabilized inference or widens to include no effect.
Three results would be decisive. The first is the gap in coverage between the calendar-adjusted baseline and our estimator as learning strength grows. The second is the cross-fitting ablation. The third is whether the test flags the outcome-driven simulations while staying near its 5% false-positive rate on the drift-only control.
References
- Post-Contextual-Bandit Inference
- Statistical Inference with M-Estimators on Adaptively Collected Data
- Policy Learning with Adaptively Collected Data
- Semiparametric Efficient Inference in Adaptive Experiments
- Demystifying Inference After Adaptive Experiments
- Timing to intubation COVID-19 patients: can we put it off until tomorrow?
- Long-Term Survival in Men With High Grade Prostate Cancer: A Comparison Between Conservative Treatment, Radiation Therapy and Radical Prostatectomy—A Propensity Scoring Approach
- Towards Regulatory-Confirmed Adaptive Clinical Trials: Machine Learning Opportunities and Solutions
- Simulation-Based Inference for Adaptive Experiments