Abstract
In this paper, we explore ways of combining experimental data and non-experimental methods to estimate the differential effects of components of training programs. We show how data from a multi-site experimental evaluation in which subjects are randomly assigned to any treatment versus a control group who receives no treatment can be combined with non-experimental regression-adjustment methods to estimate the differential effects of particular types of treatments. We also devise tests of the validity of using the latter methods. We use these methods and tests to re-analyze data from the MDRC Evaluation of California's Greater Avenues to Independence (GAIN) program.
Introduction: The "Riverside Miracle" and Welfare Reform
In the 1990s, California’s Greater Avenues to Independence (GAIN) program became a national model for welfare reform. Initial evaluations by the MDRC suggested that Riverside County, which emphasized Labor Force Attachment (LFA) (getting people into any job quickly), was far more successful than counties emphasizing Human Capital Development (HCD) (education and vocational training). This finding, often called the "Riverside Miracle," led to a nationwide policy shift toward "work-first" strategies.
However, these early evaluations suffered from two major issues:
- Confounding Variables: Riverside’s success might have been due to its unique local economy or the specific demographics of its participants, rather than the LFA strategy itself.
- Short-Term Bias: HCD takes time to complete. Evaluating a program after only three years naturally favors "quick-fix" employment strategies over long-term skill building.
Methodology: Validating Non-Experimental Methods
The authors propose a hybrid approach that combines experimental and non-experimental techniques to isolate the treatment effect of training components from site-specific effects:
- Non-Experimental Regression Adjustment: Parametric models are used to adjust for differences in participant backgrounds and local economic conditions across different counties.
- Validation via Experimental Control Groups: This is the paper's primary methodological contribution. Since the original GAIN study had a randomly assigned control group (who received no training) in every county, the authors use these groups to test their model. If the regression model correctly adjusts for site differences, it should show zero difference between the control groups across counties. If it doesn't, the model is biased.
- Difference-in-Differences (DID): A DID estimator compares the "gains" of treatment groups relative to their respective local control groups, further isolating the impact of the specific training type (LFA vs. HCD).
Data and Long-Term Tracking
The study utilizes data from the MDRC GAIN Evaluation across four California counties: Alameda, Los Angeles, Riverside, and San Diego. This is supplemented by administrative records, including California’s Unemployment Insurance (UI) Base Wage files for earnings and the Medi-Cal Eligibility Data System (MEDS) for welfare participation. Crucially, the authors extend the follow-up period to nine years, far beyond the original three-to-five-year studies, allowing for an analysis of long-term human capital returns.
Key Findings: The Human Capital Development "Crossover"
The re-analysis yields several critical insights that challenge the "work-first" consensus:
- The "Miracle" was partly a Market Effect: Much of Riverside’s perceived advantage was actually due to its relatively stronger local labor market during the early 1990s recovery compared to counties like Los Angeles.
- The HCD "Crossover": There is a clear temporal trade-off. LFA produces immediate employment gains because participants aren't "sidelined" by classes. However, HCD participants eventually catch up and surpass LFA participants in employment stability.
- Long-Term Reversal: While LFA showed a strong positive differential effect in years 1-3, the trend reversed by years 7-9. HCD participants had higher employment rates. When adjusting for local economic conditions, the long-term negative differential effect of LFA (relative to HCD) on employment was nearly double the size of unadjusted estimates and statistically significant.
Conclusion and Policy Implications
The study demonstrates that simply comparing counties without adjusting for participant characteristics leads to biased and likely false estimates of program effectiveness. If the goal of welfare-to-work is long-term self-sufficiency rather than a "quick fix," the policy preference for work-first programs over skill development needs a serious rethink. Human capital investments, while slower to manifest, yield significantly better employment outcomes in the long run.
Summary updated