Abstract
Researchers increasingly have access to two types of data: (i) large observational datasets where treatment (e.g., class size) is not randomized but several primary outcomes (e.g., graduation rates) and secondary outcomes (e.g., test scores) are observed and (ii) experimental data in which treatment is randomized but only secondary outcomes are observed. We develop a new method to estimate treatment effects on primary outcomes in such settings. We use the difference between the secondary outcome and its predicted value based on the experimental treatment effect to measure selection bias in the observational data. Controlling for this estimate of selection bias yields an unbiased estimate of the treatment effect on the primary outcome under a new assumption that we term latent unconfoundedness, which requires that the same confounders affect the primary and secondary outcomes. Latent unconfoundedness weakens the assumptions underlying commonly used surrogate estimators. We apply our estimator to identify the effect of third grade class size on students outcomes. Estimated impacts on test scores using OLS regressions in observational school district data have the opposite sign of estimates from the Tennessee STAR experiment. In contrast, selection-corrected estimates in the observational data replicate the experimental estimates. Our estimator reveals that reducing class sizes by 25% increases high school graduation rates by 0.7 percentage points. Controlling for observables does not change the OLS estimates, demonstrating that experimental selection correction can remove biases that cannot be addressed with standard controls.
The Data Trade-off in Policy Research
In social science and policy evaluation, researchers often face a fundamental trade-off between the internal validity of randomized controlled trials (RCTs) and the long-term scope of observational administrative data.
- Experimental Data: RCTs provide clean causal links but are typically small-scale and short-lived. For instance, we might know if a program improved 3rd-grade test scores, but we rarely have the resources to track those same individuals for decades to see if they graduated from college.
- Observational Data: Large-scale administrative records can track individuals for decades, covering long-term "primary" outcomes like graduation or lifetime earnings. However, these datasets are prone to selection bias because treatment is not randomized. In education, for example, schools might intentionally place struggling students in smaller classes, leading a naive analysis to suggest that small classes cause lower achievement.
The Experimental Selection Correction (ESC) Estimator
To bridge this gap, we introduce the Experimental Selection Correction (ESC) estimator. This method uses a small-scale experiment to "calibrate" the selection bias present in a large-scale observational dataset. The process follows three steps:
- Identify the True Short-term Effect: Use experimental data to find the unbiased effect of the treatment on a secondary, short-term outcome (e.g., 3rd-grade test scores).
- Calculate the Selection Residual: In the observational data, calculate the difference between an individual's actual short-term outcome and what the experiment predicts it should have been. This "selection residual" serves as a direct measure of hidden selection bias—the unobserved factors that influenced treatment assignment.
- Correct the Long-term Estimate: Regress the long-term primary outcome (e.g., graduation) on the treatment in the observational data while controlling for the selection residual calculated in the previous step.
Latent Unconfoundedness
The validity of this approach rests on the assumption of Latent Unconfoundedness. This assumes that the unobserved factors biasing short-term results (such as parental involvement or student motivation) are the same factors biasing long-term results. By "subtracting out" the bias from the intermediate test scores, we effectively clean the long-term graduation data as well.
Application: Class Size and Student Outcomes
We validated the ESC estimator using two landmark datasets: the Project STAR experiment (randomized class sizes in Tennessee) and New York City Administrative Records (observational data on 1.76 million children).
In the raw NYC observational data, smaller classes were associated with lower graduation rates—the "wrong sign" problem caused by the system funneling at-risk students into smaller classes. Standard demographic controls failed to fix this bias. However, the ESC estimator successfully flipped these estimates, replicating the known experimental effects on intermediate scores and providing a credible estimate for long-term outcomes.
Key Findings
- Graduation Impact: Reducing class size by 25% (approximately 7 students) increases high school graduation rates by 0.7 percentage points.
- Beyond Surrogacy: Unlike traditional surrogate models, the ESC estimator does not require the treatment to work only through test scores. It accounts for direct effects on long-term outcomes that might bypass intermediate metrics (e.g., improvements in non-cognitive skills).
- Addressing Fade-out: While test score gains from small classes often "fade out" by middle school, the graduation benefits persist. The ESC estimator correctly identifies this long-term value where surrogate models based on middle-school scores would fail.
Conclusion
The ESC estimator demonstrates that experimental selection correction can remove deep-seated biases that standard controls cannot address. By combining the internal validity of experiments with the scale of administrative data, researchers can produce more reliable estimates of the long-term impacts of social and educational interventions.
Summary updated