Abstract
Researchers often face a dilemma: use small, unbiased experimental data or large, biased observational data. This paper introduces the Experimental Selection Correction (ESC) estimator, which leverages small-scale randomized experiments to anchor and "clean" large-scale observational datasets. By using a shared secondary outcome to measure latent selection bias, the ESC provides accurate estimates for long-term primary outcomes that were never measured in the original experiment.
Introduction
Causal inference in social sciences typically relies on two types of data, each with significant drawbacks. Experimental data, such as randomized controlled trials (RCTs), offer unbiased estimates but are often limited in scale and lack long-term follow-up. Conversely, observational administrative records provide massive sample sizes and long-term outcomes but are frequently compromised by selection bias. For instance, in education, "at-risk" students might be intentionally placed in smaller classes, making small classes appear ineffective or even harmful in raw observational data.
The Limitations of Surrogacy
A common approach to bridge this gap is the "surrogate" model. This method assumes that a treatment's effect on a long-term outcome (e.g., graduation) is entirely mediated through a short-term "surrogate" outcome (e.g., test scores). However, this assumption is often too restrictive. If a treatment also influences "soft skills" or other unmeasured factors that independently affect long-term success, the surrogate approach will yield biased results.
The Experimental Selection Correction (ESC) Estimator
The ESC estimator relaxes these assumptions by introducing a "latent unconfoundedness" framework. It allows the treatment to have a direct effect on the primary outcome that does not pass through the surrogate, while accounting for hidden factors that influence both short-term and long-term results.
The ESC method operates in three primary steps:
- The Anchor: Use experimental data to determine the true causal effect of a treatment on a secondary (surrogate) outcome (e.g., how class size actually affects 3rd-grade test scores).
- The Bias Measure: In the observational data, calculate the "residual"—the difference between the observed secondary outcome and the outcome predicted by the experimental effect. This residual serves as a proxy for unobserved selection bias.
- The Correction: Regress the primary long-term outcome (graduation) on the treatment in the observational data while controlling for this bias residual.
Empirical Validation: Project STAR and NYC Schools
The authors validate the ESC estimator using two landmark datasets. Project STAR provided experimental data on class sizes and test scores in Tennessee but lacked reliable long-term graduation data. New York City administrative records provided observational data for 1.76 million children, including class sizes, test scores, and high school graduation rates.
In the NYC data, standard OLS regressions produced a counterintuitive result: smaller classes appeared to reduce graduation rates by 1.76 percentage points. This negative correlation is a classic example of selection bias, where lower-performing students are prioritized for smaller classes.
Key Results
By applying the ESC estimator, the researchers were able to "flip the script" on this bias. The corrected estimate showed that reducing class sizes by 25% in the 3rd grade actually increases the probability of high school graduation by 0.7 percentage points.
The model's validity was further confirmed through a "fadeout" test. Using 3rd-grade scores to correct the NYC data, the researchers checked if the corrected estimates for 4th through 8th-grade scores matched the experimental results from Project STAR. The ESC successfully captured the well-documented "fadeout" effect, where initial test score gains diminish over time, matching the experimental ground truth.
Conclusion
The ESC estimator represents a significant advancement in causal inference, providing a robust method for combining the internal validity of experiments with the external validity and scale of observational data. By moving beyond the strict requirements of surrogacy models, it allows researchers to uncover true causal relationships in complex, real-world datasets. Future work may extend this framework to handle multiple intermediate variables or further relax the underlying assumptions of latent unconfoundedness.
Summary updated