In his seminal 1969 paper, Donald T. Campbell lays out a manifesto for the "experimental society." He argues that social reforms—from traffic laws to job training programs—should be treated as scientific experiments. Rather than committing to a single "cure-all" solution, policymakers should adopt an iterative approach: try a program, rigorously evaluate its impact, and then retain, modify, or discard it based on evidence.
TL;DR
Campbell proposes that social reforms be treated as experiments to determine their true effectiveness. He introduces a toolkit of quasi-experimental designs to help "experimental administrators" navigate the political and statistical pitfalls of evaluating public policy in the real world.
What Problem Does This Paper Address?
The paper tackles the gap between the high-minded promises of social legislation (like the "Great Society" programs) and the lack of hard evidence regarding their actual impact. Campbell identifies a major political hurdle: the "trapped administrator." Because politicians often advocate for specific reforms as certain successes, any honest evaluation that shows failure becomes a personal and political liability. This leads to "obfuscation and self-deception," where administrators rely on "grateful testimonials" rather than data. Campbell seeks to provide the methodological and political framework to move toward an "experimental" stance where the focus is on solving the problem, not defending a specific (and potentially failing) solution.
Methods
Campbell introduces a taxonomy of quasi-experimental designs—methods that allow for causal inference when the "gold standard" of randomized controlled trials (RCTs) is not feasible. Key designs include:
- Interrupted Time-Series Design: Analyzing a long-term trend before and after a reform is implemented.
- Control Series Design: Improving the time-series by comparing the target group with a similar, non-equivalent control group (e.g., a neighboring state).
- Regression Discontinuity Design: A clever way to evaluate programs with scarce resources. If a benefit is given based on a strict cutoff (like a test score), you can compare individuals just above the threshold to those just below it, effectively mimicking a random assignment at the margin.
He also provides a checklist of nine threats to internal validity, such as "history" (external events), "maturation" (natural trends), and "regression artifacts" (statistical anomalies).
Datasets
The paper uses several historical case studies to illustrate these methods:
- Connecticut Traffic Fatalities (1951–1959): Evaluating Governor Ribicoff’s 1955 crackdown on speeding.
- Chicago Crime Statistics (1942–1962): Examining the impact of police reorganization under Orlando Wilson.
- German Divorce Rates (1881–1914): Analyzing the effect of legal changes in 1900.
- Job Training Corps & National Merit Scholarships: Used as hypothetical or emerging examples for regression discontinuity.
Key Insights and Ideas
- The "Trapped" vs. "Experimental" Administrator: The most important shift is political. Administrators should advocate for the importance of the problem rather than the certainty of the answer. This allows them to pivot to "Policy B" if "Policy A" fails without losing face.
- Regression to the Mean is a Silent Killer: Programs are often implemented after a "worst-ever" year. Statistically, the next year is likely to be better regardless of the reform. Campbell shows how a simple before-and-after comparison can trick us into seeing success where there is only a natural statistical return to the average.
- Instrumentation Changes: "New brooms" (new administrators) often reform record-keeping at the same time they reform policy. This "confounds" the results—did crime go up, or did we just get better at counting it?
- Multiple Measures: To avoid "irrelevant responsiveness" (where people "game" a single metric), administrators should use multiple, independent indicators of success.
Key Results
Campbell’s re-analysis of the Connecticut speeding crackdown provides a masterclass in skepticism. While the Governor claimed a 12.3% reduction in deaths in 1956, Campbell’s extended time-series shows that 1955 was an exceptionally high outlier.
In Chicago, police reform led to a massive increase in reported petty larcenies. This wasn't a crime wave; it was an instrumentation effect. The new administration insisted on honest bookkeeping, recording crimes that previous officers had ignored to make their precincts look better.
Limitations and Future Work
Campbell acknowledges that quasi-experiments are always vulnerable to "plausible rival hypotheses." He stresses the need for external validity—knowing if a result from one city will work in another. He also notes the difficulty of maintaining "sharp cutoffs" in regression discontinuity designs, as political patronage often "fuzzes" the boundaries. Finally, he calls for the creation of "social indicators" and "data banks" (an early vision for modern data-driven policy) to make these evaluations easier and more transparent.
Summary updated