Long-Term Effect Estimation via Surrogate Index Data Fusion

This direction is merged into another direction: unobserved confounding (unmeasured confounding). Its last version stays readable and citable here.

Version 1 of 126 papers42 conceptsfrozen 2026-10-04

In fields ranging from medicine to digital economics, the most important outcomes often take years to manifest. A medical researcher wants to know if a new drug increases five-year survival rates; an economist wants to know if a job training program increases lifetime earnings; a software engineer wants to know if a UI change increases annual user retention.

However, waiting years for results is often impractical. The standard solution is to use surrogates—short-term proxies like 3-month biomarkers, early-career employment, or 1-week click-through rates. But relying on a single surrogate is risky. As Susan Athey, Guido Imbens, Raj Chetty, and Hyunseung Kang argue in Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index, a treatment might improve a specific proxy without actually helping the long-term goal—a phenomenon known as the "surrogate paradox."

The research direction emerging from these papers focuses on data fusion: combining short-term experimental data with long-term observational data to build a robust "Surrogate Index" that can predict the future without waiting for it.

The Core Mechanism: Bridging Two Worlds

The fundamental challenge of long-term causal inference is that we rarely have a single dataset that is both experimental (randomized) and long-term.

  1. The Experiment: Provides a clean, unbiased estimate of how a treatment affects short-term proxies ($S$), but doesn't last long enough to see the final outcome ($Y$).
  2. The Observational Data: Contains historical records of both proxies ($S$) and long-term outcomes ($Y$), but the "treatment" in this data wasn't randomized, making it biased.

The Surrogate Index acts as a bridge. As introduced by Athey et al. in The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely, researchers use the observational data to learn the relationship between $S$ and $Y$. They then apply this learned mapping to the experimental data to estimate the treatment's effect on $Y$. By using a combination of multiple surrogates rather than just one, the index becomes significantly more precise and harder to "game."

The Surrogacy Assumption and Its Failures

For this bridge to hold, researchers rely on the surrogacy assumption (or exclusion restriction). This assumes that the treatment's effect on the long-term outcome is entirely mediated by the observed surrogates. In other words, if you know the surrogates, knowing whether the person received the treatment provides no additional information about their long-term outcome.

This assumption is the "Achilles' heel" of the field. Several papers highlight how it can fail:

From One-Time Shocks to Continuous Treatments

Traditional surrogate methods often treat the treatment as a "one-time shock." However, many real-world interventions are persistent—like a chronic medication or a permanent change to a website's ranking algorithm.

Allen Tran, Aurélien Bibaut, and Nathan Kallus argue in Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments that standard surrogate indices underestimate the impact of these treatments because they ignore the compounding effect of continued exposure after the surrogate is measured.

To solve this, the field is increasingly borrowing from Offline Reinforcement Learning (ORL). By framing the problem as a Markov Decision Process, researchers can model the temporal dynamics of how treatments and surrogates interact over time. For instance, Yiheng Duan and Lihong Li use an RL framework in A Reinforcement Learning Approach to Estimating Long-term Treatment Effects to filter out "noise" like seasonality, while Yuta Saito and colleagues at Spotify introduce LOPE (Long-term Off-Policy Evaluation) to decompose rewards into "surrogate" and "action" effects, allowing for more robust evaluation in complex digital environments.

Learning Better Representations

Finally, the research is moving away from manually selecting surrogates toward representation learning. Instead of picking "clicks" or "test scores," models like the LTEE framework by Lu Cheng and Ruocheng Guo (Long-Term Effect Estimation with Surrogate Representation) use deep sequential models to learn latent representations of surrogates.

Similarly, Weilin Chen and Ruichu Cai leverage Optimal Transport in Estimating Long-term Heterogeneous Dose-response Curve to align the distributions of experimental and observational data, ensuring that the "bridge" built between the two datasets is as accurate as possible, even when unobserved factors are at play.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_long_term_effect_estimation_via_surrogat_2f88c41e,
  title = {Long-Term Effect Estimation via Surrogate Index Data Fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/long-term-effect-estimation-via-surrogate-index-data-fusion/rdr_2f88c41e7e2447f58900f9ddd0780714},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_long_term_effect_estimation_via_surrogat_2f88c41e_v1,
  title = {Long-Term Effect Estimation via Surrogate Index Data Fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/long-term-effect-estimation-via-surrogate-index-data-fusion/rdr_2f88c41e7e2447f58900f9ddd0780714/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}