In fields ranging from medicine to digital economics, the most important outcomes often take years to manifest. A medical researcher wants to know if a new drug increases five-year survival rates; an economist wants to know if a job training program increases lifetime earnings; a software engineer wants to know if a UI change increases annual user retention.
However, waiting years for results is often impractical. The standard solution is to use surrogates—short-term proxies like 3-month biomarkers, early-career employment, or 1-week click-through rates. But relying on a single surrogate is risky. As Susan Athey, Guido Imbens, Raj Chetty, and Hyunseung Kang argue in Estimating Treatment Effects using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate Index, a treatment might improve a specific proxy without actually helping the long-term goal—a phenomenon known as the "surrogate paradox."
The research direction emerging from these papers focuses on data fusion: combining short-term experimental data with long-term observational data to build a robust "Surrogate Index" that can predict the future without waiting for it.
The Core Mechanism: Bridging Two Worlds
The fundamental challenge of long-term causal inference is that we rarely have a single dataset that is both experimental (randomized) and long-term.
- The Experiment: Provides a clean, unbiased estimate of how a treatment affects short-term proxies ($S$), but doesn't last long enough to see the final outcome ($Y$).
- The Observational Data: Contains historical records of both proxies ($S$) and long-term outcomes ($Y$), but the "treatment" in this data wasn't randomized, making it biased.
The Surrogate Index acts as a bridge. As introduced by Athey et al. in The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely, researchers use the observational data to learn the relationship between $S$ and $Y$. They then apply this learned mapping to the experimental data to estimate the treatment's effect on $Y$. By using a combination of multiple surrogates rather than just one, the index becomes significantly more precise and harder to "game."
The Surrogacy Assumption and Its Failures
For this bridge to hold, researchers rely on the surrogacy assumption (or exclusion restriction). This assumes that the treatment's effect on the long-term outcome is entirely mediated by the observed surrogates. In other words, if you know the surrogates, knowing whether the person received the treatment provides no additional information about their long-term outcome.
This assumption is the "Achilles' heel" of the field. Several papers highlight how it can fail:
- Direct Effects: The treatment might affect the long-term outcome through a path that bypasses the surrogates. Susan Athey and colleagues address this in The Experimental Selection Correction Estimator, proposing the ESC estimator to "clean" observational bias by using small experiments as anchors, even when surrogacy is imperfect.
- Unobserved Confounding: Hidden factors might influence both the surrogates and the final outcome. Nathan Kallus, Xiaojie Mao, and Yuhao Wang tackle this in Long-term Causal Inference Under Persistent Confounding via Data Combination, using the sequential structure of multiple proxies to account for these "persistent" hidden biases.
- Information Leakage: If the treatment influences latent traits not captured by the index, the model will suffer from "effect leakage," leading to biased predictions Aurélien Bibaut et al. in Long-Term Causal Inference with Imperfect Surrogates using Many Weak Experiments, Proxies, and Cross-Fold Moments.
From One-Time Shocks to Continuous Treatments
Traditional surrogate methods often treat the treatment as a "one-time shock." However, many real-world interventions are persistent—like a chronic medication or a permanent change to a website's ranking algorithm.
Allen Tran, Aurélien Bibaut, and Nathan Kallus argue in Inferring the Long-Term Causal Effects of Long-Term Treatments from Short-Term Experiments that standard surrogate indices underestimate the impact of these treatments because they ignore the compounding effect of continued exposure after the surrogate is measured.
To solve this, the field is increasingly borrowing from Offline Reinforcement Learning (ORL). By framing the problem as a Markov Decision Process, researchers can model the temporal dynamics of how treatments and surrogates interact over time. For instance, Yiheng Duan and Lihong Li use an RL framework in A Reinforcement Learning Approach to Estimating Long-term Treatment Effects to filter out "noise" like seasonality, while Yuta Saito and colleagues at Spotify introduce LOPE (Long-term Off-Policy Evaluation) to decompose rewards into "surrogate" and "action" effects, allowing for more robust evaluation in complex digital environments.
Learning Better Representations
Finally, the research is moving away from manually selecting surrogates toward representation learning. Instead of picking "clicks" or "test scores," models like the LTEE framework by Lu Cheng and Ruocheng Guo (Long-Term Effect Estimation with Surrogate Representation) use deep sequential models to learn latent representations of surrogates.
Similarly, Weilin Chen and Ruichu Cai leverage Optimal Transport in Estimating Long-term Heterogeneous Dose-response Curve to align the distributions of experimental and observational data, ensuring that the "bridge" built between the two datasets is as accurate as possible, even when unobserved factors are at play.