The Unreasonable Effectiveness of Algorithms

Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan

Abstract

We calculate the social return on algorithmic interventions (specifically their Marginal Value of Public Funds) across multiple domains of interest to economists—regulation, criminal justice, medicine, and education. Though these algorithms are different, the results are similar and striking. Each one has an MVPF of infinity: not only does it produce large benefits, it provides a "free lunch." We do not take these numbers to mean these interventions ought to be necessarily scaled, but rather that much more R&D should be devoted to developing and carefully evaluating algorithmic solutions to policy problems.

Introduction: The Ranking Problem in Public Policy

In the current discourse surrounding Artificial Intelligence, much of the focus remains on large language models and consumer-facing applications. However, a quieter revolution is occurring in the "fundamentals" of public policy. Human decision-makers—judges, doctors, and inspectors—are tasked daily with complex "ranking problems." They must decide which defendants are high-risk, which patients require intensive testing, and which workplaces are most likely to suffer accidents.

Evidence suggests that human judgment in these domains is often "flat." Because of cognitive biases, heuristics, and noise, humans frequently misrank cases, leading to significant social inefficiencies. We might detain low-risk individuals while releasing high-risk ones, or order expensive medical tests for patients with negligible risk. Furthermore, traditional policy interventions, such as reducing class sizes or increasing police presence, are notoriously difficult to scale because they rely on human talent—a scarce and variable resource.

Methodology: The Marginal Value of Public Funds (MVPF)

To evaluate the efficacy of algorithmic interventions, we employ the Marginal Value of Public Funds (MVPF) framework. The MVPF is a standard metric in public finance used to compare the cost-effectiveness of different policies. It is defined as the ratio of the net social benefit ($\Delta W$) to the net cost to the government ($\Delta E - \Delta C$):

$$MVPF = \frac{\Delta W}{\Delta E - \Delta C}$$

Where:

  • $\Delta W$ represents the social benefit (e.g., the value of freedom for a defendant or health gains for a patient).
  • $\Delta E$ is the upfront government expenditure required to build and deploy the algorithm.
  • $\Delta C$ represents the government savings generated by the intervention (e.g., reduced incarceration costs or fewer unnecessary medical procedures).

A unique phenomenon occurs when the savings ($\Delta C$) exceed the implementation costs ($\Delta E$). In these cases, the denominator becomes negative, resulting in an MVPF of infinity. This signifies a "free lunch": the policy improves social welfare while simultaneously saving the government money.

Case Studies: Algorithmic Efficiency Across Domains

We analyzed four distinct case studies based on existing randomized controlled trials (RCTs) and policy simulations. The results were remarkably consistent across all domains.

1. Criminal Justice: Pretrial Release

Using data from New York City and Kentucky, researchers found that algorithms could significantly improve bail decisions. An NYC-based algorithm could reduce detentions by 40% without increasing crime rates. With an estimated $4 million development cost and $34.5 million in annual jail savings, the intervention provides an MVPF of infinity.

2. Safety Regulation: OSHA Inspections

By better targeting workplace inspections using historical injury records, algorithms could avert approximately 16,000 serious injuries. The resulting increase in tax revenue from prevented lost income ($209 million) far outweighs the cost of the software, again yielding an MVPF of infinity.

3. Healthcare: Medical Testing

In the context of Medicare, algorithmic rules designed to predict heart attack risk could reduce unnecessary EKG testing by 34.7% without compromising patient outcomes. This would save the government approximately $4.8 billion per year, resulting in an MVPF of infinity.

4. Education: Course Placement

In a study involving seven colleges, algorithmic course placement saved students an average of $150 and the government $230 per student. Given an implementation cost of only $140, the program pays for itself while improving educational efficiency.

Why Algorithms Succeed: Scalability and the "Flat" Baseline

The "unreasonable effectiveness" of these tools stems from two primary factors. First, algorithms address the "too flat" ranking problem. By accurately re-ranking cases based on data-driven risk profiles, algorithms create a steeper benefit curve, significantly reducing the deadweight loss associated with human error.

Second, algorithms possess a unique scale-up advantage. Unlike human-centric programs, which often lose effectiveness when expanded to less-resourced agencies, algorithms are software. They can be replicated and run repeatedly at near-zero marginal cost without a loss in quality or "fidelity."

Challenges and Future Directions

Despite these promising results, several hurdles remain before widespread implementation is feasible:

  • Human-Algorithm Interaction: We must understand whether human decision-makers will effectively use algorithmic advice or if they will override it based on "private information" or bias.
  • Data Drift and Generalization: Algorithms must be retrained as the world changes (e.g., shifting crime patterns). Furthermore, a model built for a large urban center may not generalize to a rural context without adjustment.
  • Strategic Behavior: In adversarial environments, actors may attempt to "game" the algorithm, necessitating robust design.
  • Equity and Bias: While algorithms can reduce human bias, they can also codify and scale biases present in historical data. Careful evaluation is required to ensure efficiency does not come at the cost of fairness.

Conclusion

When compared to a library of 130 traditional policy interventions, these algorithmic solutions consistently rank in the top 15% for cost-effectiveness. The prevalence of "infinite" MVPFs suggests that we are currently under-investing in this space. Rather than blindly scaling every available tool, we should significantly increase the R&D pipeline to develop, test, and refine algorithmic solutions to our most pressing policy challenges.

Summary updated

Abstract

We calculate the social return on algorithmic interventions (specifically their Marginal Value of Public Funds) across multiple domains of interest to economists-regulation, criminal justice, medicine, and education.Though these algorithms are different, the results are similar and striking.Each one has an MVPF of infinity: not only does it produce large benefits, it provides a "free lunch."We do not take these numbers to mean these interventions ought to be necessarily scaled, but rather that much more R&D should be devoted to developing and carefully evaluating algorithmic solutions to policy problems.
BibTeX
@misc{ludwig2024unreasonable,
  author = {Jens Ludwig and Sendhil Mullainathan and Ashesh Rambachan},
  title = {{The Unreasonable Effectiveness of Algorithms}},
  year = {2024},
  doi = {10.2139/ssrn.4716690},
  howpublished = {SSRN},
}
Figure