Tracking Confounding Bias Across the Trial Window

Most randomized trials stop before the outcome that matters has happened. A six-month kidney trial ends before patients reach renal failure. A five-week exercise trial ends before body composition changes. Retrospective cohorts built from health records do follow patients for years, but treatment in them was chosen by doctors and patients, so the comparison is biased in ways no adjustment fully removes. Several methods already combine the two kinds of data. Each one rests on a single untestable assumption about how the bias at the short horizon relates to the bias at the long horizon. This note proposes a simple change. Many short trials already watch the outcome begin to unfold: cardiovascular events accumulate month by month, employment is recorded every quarter, and test scores are taken every school year. Within that window, we can measure the observational cohort's bias directly at several time points, by comparing its estimate against the randomized truth at each point. We can then see whether the bias is stable, drifting, or erratic, and extrapolate it to the long horizon. The width of the resulting interval should reflect how far we are extrapolating.

In two sentences: the randomized trial tells you how wrong the observational cohort is at month 1, 3, and 6, so fit that error curve and correct the cohort's 5-year estimate by the extrapolated error. The error curve also tells you when to refuse to give a point estimate, because it shows whether the assumption the existing methods need actually held while it could be checked.

Core Claim

Let $\tau_E(t)$ be the randomized trial's effect on the cumulative outcome at horizon $t$, observed only up to the trial's end $T_E$. Let $\tau_O(t)$ be the covariate-adjusted treated-versus-untreated contrast in the retrospective cohort, available at every horizon. For $t \le T_E$, the bias $b(t) = \tau_O(t) - \tau_E(t)$ can be estimated directly. The proposed long-horizon estimate is

$$\hat\tau(t^\ast) = \hat\tau_O(t^\ast) - \hat b(t^\ast),$$

where $\hat b(t^\ast)$ comes from a low-dimensional curve fitted to the within-window bias estimates. The interval around it is widened by a sensitivity parameter $\Lambda$. $\Lambda$ caps how much the bias curve's slope may change after $T_E$, expressed as a multiple of the largest slope change actually observed within the window.

The equi-confounding strategy of Ghassami et al. fuses Project STAR with ECLS-K by assuming unmeasured confounding biases the short- and long-term outcomes equally. That assumption is the special case $b(t)$ constant with $\Lambda = 0$. Our claim is that this assumption can be partly tested whenever the trial records the outcome at more than one horizon, and that it should be relaxed when the test fails.

The same bias-tracking logic applies to the surrogate index of Athey et al.. The surrogate index is a model, fitted on the observational cohort, that predicts the long-term outcome from short-term measurements; it is then used to fill in the missing long-term outcomes for trial participants. Its prediction error can be measured at horizons the trial observed, by building the index from earlier measurements and predicting a later in-window outcome. Its error trajectory can then be extrapolated in the same way.

The contribution is a family of fusion estimators whose key assumption is checked inside the trial window. It comes with a pre-registered rule for abstaining when the check fails, and with the first benchmark that scores fusion methods against true long-horizon randomized effects on clinical and social trials rather than digital A/B tests.

Relation to Prior Work

Two lines of work motivate and supply the setup.

The first line combines short-term surrogates with observational outcomes. The surrogate index of Athey et al. recovered the 9-year employment effect in the GAIN job-training experiment from 1.5 years of data. Zhang et al. showed that across 200 Netflix A/B tests, 14-day surrogate inferences matched 63-day reads in about 95% of cases. Both assume no unmeasured confounding between the surrogates and the outcome. We keep their imputation machinery but measure its error over time instead of assuming that error is zero.

The second line identifies long-term effects despite hidden confounding by adding structure:

  • Hu, Zhou, and Wu assume that the long-term outcome's conditional mean, given surrogates and covariates, is the same in both datasets. They build doubly robust estimators on that basis and fuse a 6-month IgA nephropathy trial with a 547-patient retrospective cohort.
  • Ghassami et al. offer equi-confounding, a bespoke instrument, and proxy variables.
  • Van Goffrier et al. build a synthetic instrument from trial residuals, valid under linear or partially linear models.
  • Bibaut et al. handle surrogates that miss part of the effect by using a library of many weak experiments.

All four use one short horizon and one long horizon. None uses the trial's repeated in-window outcome measurements to test its own assumption. Bibaut et al. also need dozens to thousands of experiments, which a single clinical trial cannot provide.

Two pieces of evidence constrain the design. The clinical surrogate failures in the file show that bias can change sign or grow after the trial window: cardiac remodeling that does not translate into survival (Sohal et al.), and glycemic gains that do not confirm prevention of complications (Huang et al.). Late toxicities are the clearest case. That rules out designs that present an extrapolated bias curve as if it were identified, and it is why the sensitivity parameter $\Lambda$ and the abstention rule are central rather than decorative. The persistence of residual confounding after propensity matching in retrospective work such as Tewari et al. is what makes $b(t)$ non-zero in the first place.

Research Program

Estimation at each horizon. We estimate $\tau_E(t)$ and $\tau_O(t)$ on a grid of horizons inside the trial window. For time-to-event outcomes, we use either cumulative incidence or restricted mean survival time. Censoring is handled by inverse probability of censoring weighting, the device the survival metalearner work uses, which reweights observed patients to stand in for those lost to follow-up. $\tau_O(t)$ is estimated with cross-fitted doubly robust adjustment for measured covariates, so that $b(t)$ captures only unmeasured confounding plus any mismatch between the two populations. The joint covariance of the $\hat b(t)$ estimates is obtained by bootstrap, resampling patients within each dataset.

Extrapolating the bias curve. We fit three nested curve families: constant, linear in log-time, and a monotone spline with few knots. The family is chosen by in-window leave-last-horizon-out prediction: fit on early horizons and check whether the curve predicts the bias at the last observed horizon.

Surrogate variant. The same procedure is applied to the surrogate index's imputation error, giving a second estimator that targets cases where the surrogates absorb most of the confounding.

Abstention rule. We fix in advance a rule that replaces the point estimate with a sensitivity bound when the curve is unstable. The trigger is that the leave-last-horizon-out residual exceeds its bootstrap 90% band.

Theory. We will show two results. First, under a bias curve with slope changes bounded by $\Lambda$ times the in-window maximum, the interval has nominal coverage. Second, the method reduces to equi-confounding when the in-window slope is zero. We do not claim coverage when bias jumps after $T_E$, and we test that failure mode on purpose.

Baselines

Naive references. The first is the trial's own effect at its last horizon, $\tau_E(T_E)$, reported as if it were the long-term effect. This is what a short trial publishes today. The second is the retrospective cohort's adjusted long-horizon contrast $\tau_O(t^\ast)$ alone, which is the standard propensity-weighted cohort study. Both are run on exactly the same data splits as every other method.

Surrogate index. We use the method of Athey et al., in both linear and gradient-boosted versions, using the same surrogates and covariates. On GAIN, its reported estimate of the 9-year employment effect was 6.1% against a 6.4% experimental benchmark, with 35% smaller standard errors than direct measurement. Our method must match that bias on GAIN with comparable interval width, and must beat it on the clinical benchmarks where we inject confounding between surrogate and outcome.

Strongest structured competitors. These are the equi-confounding and proxy estimators of Ghassami et al., the doubly robust estimator of Hu et al., and the instrumental front-door estimator of Van Goffrier et al.. The last is run from its released code. On semi-synthetic International Stroke Trial data, Van Goffrier et al. report a 20 to 40% bias reduction over standard baselines. No reported result was found on Lacuna for Hu et al.'s numerical estimates on the IgA nephropathy data, or for Ghassami et al.'s numerical estimate on STAR/ECLS-K.

Fairness and the winning bar. All methods receive the same nuisance learners, the same cross-fitting folds, and an equal hyperparameter search budget. Beating these competitors means three things together on held-out long horizons: lower absolute bias, coverage at or above 90% at nominal 90%, and a higher rate of correct sign or launch decision.

Ablations. Two ablations isolate the key ingredient. The first is our estimator with the bias curve forced constant, which is equi-confounding applied to the early outcome. The second uses only the final in-window horizon. If the full method does not beat its constant-curve ablation in the scenarios where bias truly drifts, the multi-horizon idea has failed.

Why not just…?

Why not just assume equi-confounding, as Ghassami et al. do? Our method contains that assumption as a special case. The in-window slope test reports when the data are consistent with it, and in those cases our estimator collapses to theirs at little cost in variance. The open question is how often clinical bias curves are flat. The truncation benchmark below answers that directly. If they are nearly always flat, the honest conclusion is that equi-confounding suffices, and our contribution becomes the diagnostic rather than the estimator.

Why not just use a flexible surrogate index? A more expressive model of the outcome given the surrogates fits the observational cohort better. It does nothing about confounding between surrogates and outcome. Bibaut et al. show that under such confounding, surrogate-index confidence interval coverage collapses toward zero as samples grow. The GAIN success may reflect a setting with little such confounding. Our benchmark varies that confounding and runs the boosted surrogate index as a baseline, so the comparison is direct.

Why not just run a careful propensity-matched retrospective study? Matching balances only recorded variables. Tewari et al. concede that unmeasured confounders may still bias their survival comparison. Our method uses exactly that study's estimate, but subtracts a bias that the trial lets us measure. Where measured covariates do remove all confounding, $\hat b(t)$ will be near zero, and the two approaches will agree.

Why not just report a sensitivity analysis without the bias curve? Unanchored sensitivity bounds are usually so wide that they cannot inform decisions. Anchoring $\Lambda$ to observed in-window drift is what makes the bound informative. The real threat is a bias that changes only after the trial ends, such as late toxicity or a treatment effect that bypasses the surrogates. No in-window data can detect that. We therefore include adversarial scenarios of this kind and report how much coverage is lost, rather than claiming robustness.

Proposed Experiments

Truncation benchmark: construction. We take long-follow-up randomized trials where the long-horizon effect is known: SPRINT (n=9,206), ACCORD (n=4,711), GAIN quarterly employment over 9 years, and the International Stroke Trial with its 6-month outcomes. In each, we censor the trial at an early $T_E$; examples are 6 or 12 months for SPRINT and ACCORD, and 6 quarters for GAIN, following Athey et al. We then build a retrospective-style cohort from held-out participants. In that cohort, treatment is reassigned by a selection rule that depends on recorded covariates plus one withheld covariate, which plays the role of the unmeasured confounder.

The withheld covariate's effect on the outcome is varied across four scenarios: constant over time, growing, shrinking, and jumping after $T_E$. Only the jump scenario should break our method. The semi-synthetic confounding is our own choice, so we report results for each scenario separately and never pooled.

Truncation benchmark: scoring and pass criterion. Metrics are absolute bias against the full-follow-up randomized effect, 90% interval coverage and width, sign-correctness, and abstention rate. The decisive check has two parts. In the drifting scenarios, the full method must reduce bias against both the constant-curve ablation and the surrogate index while holding coverage. In the jump scenario, the abstention rule must trigger, or $\Lambda$-widened coverage must degrade gracefully. Failure on the first part means the idea does not work.

Real fusion case. We revisit the Project STAR and ECLS-K fusion of Ghassami et al. STAR records scores at kindergarten and grades 1, 2, and 3. That gives four in-window horizons at which the bias of the ECLS-K estimate against the experiment can be measured before extrapolating to grade 8. There is no ground truth here. The checks are whether the in-window bias curve is flat, which would support their equi-confounding estimate, and how much our interval differs from theirs.

Secondary clinical case. If the IgA nephropathy data used by Hu et al. are accessible, we will measure in-window proteinuria and early kidney-function bias and compare the extrapolated 3-, 4-, and 5-year renal failure estimates with their reported direction of benefit.