Abstract
When governments contemplate implementing social assistance or job training programs, they must assess likely effectiveness before committing resources. Often, policy makers rely on data from similar programs implemented in different time periods or localities. This paper investigates the feasibility of this "transferability," using data from the Job Training Partnership Act (JTPA) and Work Incentive (WIN) demonstrations. By examining whether results from one site can predict outcomes in another, the study explores the conditions under which external validity holds and identifies the primary obstacles to generalizing social program evaluations.
The Problem of External Validity
In the realm of policy evaluation, researchers traditionally prioritize internal validity—the certainty that a specific program caused the observed outcomes in a specific study. However, for policy makers, external validity is often more pressing: will a program that succeeded in Baltimore in 1982 yield similar results in San Diego in 1985?
This transferability problem is complicated by two primary factors:
- Population Heterogeneity: The demographic makeup and economic conditions of a new location may differ significantly from the original study site.
- Treatment Heterogeneity: Unlike a standardized medical trial, "job training" is not a uniform intervention. Programs vary in their emphasis on classroom education, vocational training, or immediate job search assistance.
Methodological Framework
The authors formalize the transferability problem using a potential outcomes framework, introducing the concept of Unconfounded Location. This assumption posits that if researchers control for a sufficiently rich set of observed characteristics—such as age, education, and detailed pre-program earnings history—the specific location itself should not bias the predicted outcome.
To test this, the researchers utilized data from four randomized Work Incentive (WIN) demonstration programs from the mid-1980s:
- Arkansas (WORK): A low-cost program ($118/trainee) focused on job search.
- Virginia (ESP): A mid-cost program ($430/trainee).
- Baltimore (Options): A high-cost program ($953/trainee) with diverse services.
- San Diego (SWIM): A high-cost program ($919/trainee) requiring both job search and work experience.
The study uses San Diego as the "target" location, attempting to predict its outcomes using data from the other three sites through propensity score matching, linear regression (OLS), and gain-score analysis. Because all four sites were randomized experiments, the authors could compare their "predictions" against the actual experimental "ground truth."
Key Insights
The Predictability of the Control Group
The study's most striking finding is that the outcomes for the control group (those who did not receive training) are remarkably predictable across different cities. Once adjusted for prior earnings history, a non-trainee in Baltimore serves as an excellent proxy for a non-trainee in San Diego. The prediction error for first-year earnings in the control group was approximately $140, a figure that was statistically insignificant. This suggests that the "Unconfounded Location" assumption holds well for baseline economic behavior.
The "Black Box" of Training
While the control group was predictable, the trainee group was not. Even after adjusting for population demographics and local economic conditions, the models failed to accurately predict the earnings of those who underwent training in San Diego. Prediction errors for trainees ranged between $290 and $320 and were statistically significant.
This discrepancy reveals that population differences are not the primary hurdle to external validity; rather, the hurdle is the variation in the "treatment" itself. Because the specific services offered in San Diego differed from those in Arkansas or Virginia, the experience of a trainee in one city was fundamentally different from a trainee in another.
The Importance of Longitudinal Data
The research emphasizes that simple demographic data (age, race, education) is insufficient for cross-site predictions. Accurate modeling requires detailed longitudinal data on participants' earnings before the program. Furthermore, the study found that "strength in numbers" matters: combining data from multiple previous locations yielded significantly more accurate predictions for the target site than relying on any single "similar" city.
Conclusion and Policy Implications
The findings suggest that while we can effectively predict how a population will perform without an intervention, predicting the impact of the intervention itself remains difficult without granular data on program components.
For program evaluations to be truly useful for future policy-making, researchers must move beyond binary "did it work?" assessments. They must record the specific "recipe" of the treatment—hours of coaching, types of classroom instruction, and vocational specifics. Only by standardizing the reporting of these program components can we move toward a reliable science of predicting social program efficacy across different contexts.
Summary updated