Predicting the Efficacy of Future Training Programs Using Past Experiences

V. Joseph Hotz, Guido W. Imbens, Julie Holland Mortimer1999 · DOI

Abstract

When governments contemplate implementing social assistance or job training programs, they must assess likely effectiveness before committing resources. Often, policy makers rely on data from similar programs implemented in different time periods or localities. This paper investigates the feasibility of this "transferability," using data from the Job Training Partnership Act (JTPA) and Work Incentive (WIN) demonstrations. By examining whether results from one site can predict outcomes in another, the study explores the conditions under which external validity holds and identifies the primary obstacles to generalizing social program evaluations.

The Problem of External Validity

In the realm of policy evaluation, researchers traditionally prioritize internal validity—the certainty that a specific program caused the observed outcomes in a specific study. However, for policy makers, external validity is often more pressing: will a program that succeeded in Baltimore in 1982 yield similar results in San Diego in 1985?

This transferability problem is complicated by two primary factors:

  1. Population Heterogeneity: The demographic makeup and economic conditions of a new location may differ significantly from the original study site.
  2. Treatment Heterogeneity: Unlike a standardized medical trial, "job training" is not a uniform intervention. Programs vary in their emphasis on classroom education, vocational training, or immediate job search assistance.

Methodological Framework

The authors formalize the transferability problem using a potential outcomes framework, introducing the concept of Unconfounded Location. This assumption posits that if researchers control for a sufficiently rich set of observed characteristics—such as age, education, and detailed pre-program earnings history—the specific location itself should not bias the predicted outcome.

To test this, the researchers utilized data from four randomized Work Incentive (WIN) demonstration programs from the mid-1980s:

  • Arkansas (WORK): A low-cost program ($118/trainee) focused on job search.
  • Virginia (ESP): A mid-cost program ($430/trainee).
  • Baltimore (Options): A high-cost program ($953/trainee) with diverse services.
  • San Diego (SWIM): A high-cost program ($919/trainee) requiring both job search and work experience.

The study uses San Diego as the "target" location, attempting to predict its outcomes using data from the other three sites through propensity score matching, linear regression (OLS), and gain-score analysis. Because all four sites were randomized experiments, the authors could compare their "predictions" against the actual experimental "ground truth."

Key Insights

The Predictability of the Control Group

The study's most striking finding is that the outcomes for the control group (those who did not receive training) are remarkably predictable across different cities. Once adjusted for prior earnings history, a non-trainee in Baltimore serves as an excellent proxy for a non-trainee in San Diego. The prediction error for first-year earnings in the control group was approximately $140, a figure that was statistically insignificant. This suggests that the "Unconfounded Location" assumption holds well for baseline economic behavior.

The "Black Box" of Training

While the control group was predictable, the trainee group was not. Even after adjusting for population demographics and local economic conditions, the models failed to accurately predict the earnings of those who underwent training in San Diego. Prediction errors for trainees ranged between $290 and $320 and were statistically significant.

This discrepancy reveals that population differences are not the primary hurdle to external validity; rather, the hurdle is the variation in the "treatment" itself. Because the specific services offered in San Diego differed from those in Arkansas or Virginia, the experience of a trainee in one city was fundamentally different from a trainee in another.

The Importance of Longitudinal Data

The research emphasizes that simple demographic data (age, race, education) is insufficient for cross-site predictions. Accurate modeling requires detailed longitudinal data on participants' earnings before the program. Furthermore, the study found that "strength in numbers" matters: combining data from multiple previous locations yielded significantly more accurate predictions for the target site than relying on any single "similar" city.

Conclusion and Policy Implications

The findings suggest that while we can effectively predict how a population will perform without an intervention, predicting the impact of the intervention itself remains difficult without granular data on program components.

For program evaluations to be truly useful for future policy-making, researchers must move beyond binary "did it work?" assessments. They must record the specific "recipe" of the treatment—hours of coaching, types of classroom instruction, and vocational specifics. Only by standardizing the reporting of these program components can we move toward a reliable science of predicting social program efficacy across different contexts.

Summary updated

Abstract

Consider a government contemplating the implementation of a training or other social assistance program. The decision to implement the program depends on the assessment of its likely e ectiveness. Often policy makers have access to data from a similar program implemented in an earlier time period or in another locality. The question arises as to how these data might be used to assess the contemplated program's likely e cacy. This situation is not uncommon. For example, the U.S. federal government's primary program for job training, the Job Training Partnership Act JTPA, is designed and administered at the local level. Thus, policy makers may wish to evaluate di erences in the e ectiveness of the local programs. In addition, the recent federal reforms to the U.S. welfare system have encouraged the development of state and local program diversity, both in the services clients receive, and in the target populations. Increasingly, states and local authorities who administer their own programs, seek to use information from other programs, conducted in the past or in other locations, to assess the likely impacts and cost-e ectiveness of these programs.
BibTeX
@misc{hotz1999predicting,
  author = {V. Joseph Hotz and Guido W. Imbens and Julie Holland Mortimer},
  title = {{Predicting the Efficacy of Future Training Programs Using Past Experiences}},
  year = {1999},
  doi = {10.3386/t0238},
}
Figure