The Unreasonable Effectiveness of Algorithms

Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan

TL;DR

Algorithmic interventions in public policy—from bail decisions to medical testing—often provide a "free lunch" for society. By calculating the Marginal Value of Public Funds (MVPF), the authors demonstrate that these tools frequently pay for themselves while delivering massive social benefits, suggesting that we are currently under-investing in algorithmic R&D.

What Problem Does This Paper Address?

The authors tackle a fundamental question in the age of AI: Are algorithms overhyped, or are they a transformative tool for public policy? While much of the "AI hype" focuses on large language models and online applications, this paper looks at the "fundamentals"—the tangible social value created when algorithms are applied to traditional economic domains like criminal justice, healthcare, and education.

The core challenge is that human decision-makers (judges, doctors, inspectors) often struggle with "ranking problems." They frequently misrank cases, leading to inefficiencies like detaining low-risk defendants while releasing high-risk ones, or ordering expensive medical tests for patients who don't need them. Furthermore, traditional policy interventions (like reducing class sizes) are difficult to scale because they rely on human talent, which is a scarce and variable resource.

Methods

The researchers use a public finance framework called the Marginal Value of Public Funds (MVPF). The MVPF is the ratio of the net benefit to society ($\Delta W$) over the net cost to the government ($\Delta E - \Delta C$).

  • $\Delta W$: Social benefit (e.g., value of freedom for a defendant, health gains for a patient).
  • $\Delta E$: Upfront government expenditure (cost to build and deploy the algorithm).
  • $\Delta C$: Government savings (e.g., reduced jail costs, fewer unnecessary medical tests).

When the savings ($\Delta C$) exceed the implementation costs ($\Delta E$), the denominator becomes negative, resulting in an MVPF of infinity. This signifies a "free lunch" where the policy improves social welfare while actually saving the government money. The paper performs "back-of-the-envelope" calculations across four case studies based on existing randomized controlled trials (RCTs) and policy simulations.

Datasets

The paper synthesizes findings from several key studies and their associated datasets:

  • Criminal Justice: New York City pretrial data and Kentucky's pretrial release system (Kleinberg et al. 2018; Albright 2023).
  • Safety Regulation: OSHA workplace injury records used to predict future accidents (Johnson et al. 2023).
  • Healthcare: Medicare patient data, including EKG results and health histories, used to predict heart attack risk (Mullainathan & Obermeyer 2022).
  • Education: Academic background and performance data from a seven-college experiment on course placement (Bergman et al. 2023).

Key Insights and Ideas

  • The "Too Flat" Ranking Problem: Humans tend to have a "flat" marginal benefit schedule because we confuse high-risk and low-risk cases. Algorithms "re-rank" these cases more accurately, creating a steeper benefit curve and significantly reducing deadweight loss.
  • The Scale-Up Advantage: Unlike human-centric programs (which often lose effectiveness when expanded to "Your Ordinary American Agency"), algorithms are software. They can be run repeatedly at near-zero marginal cost without a loss in "fidelity" or quality.
  • The Low Bar of Human Judgment: The paper argues that human judgment is often such a low bar (due to heuristics, noise, and bias) that even simple algorithms can "soar over it."
  • R&D vs. Scaling: The authors argue that while we shouldn't blindly scale every algorithm, the high MVPFs suggest we should be funneling significantly more resources into the R&D pipeline to develop and evaluate these tools.

Key Results

The results across all four domains were remarkably consistent:

  • Pretrial Release: An NYC algorithm could reduce detentions by 40% with no increase in crime. Even with a conservative $4 million build cost, the $34.5 million in annual jail savings leads to an MVPF of infinity.
  • Safety (OSHA): Better targeting of inspections could avert ~16,000 serious injuries. The resulting increase in tax revenue from prevented lost income ($209M) dwarfs the cost of the software. MVPF: Infinity.
  • Healthcare: Algorithmic rules could reduce Medicare heart attack testing by 34.7% without losing social welfare, saving the government $4.8 billion per year. MVPF: Infinity.
  • Education: Algorithmic course placement saved students $150 and the government $230 per student, with a implementation cost of only $140. MVPF: Infinity.

Compared to the "Policy Impacts" library of 130 traditional policies, these algorithmic interventions all rank in the top 15% for cost-effectiveness.

Limitations and Future Work

The authors acknowledge several critical "open questions" that must be solved before these tools can be fully realized:

  • Automation vs. Decision Aids: We don't fully know how humans will react to algorithmic advice. Will they ignore it, or will they use their own "private information" to override it effectively?
  • Data Drift: Algorithms must be retrained as the underlying world changes (e.g., changes in crime patterns or medical technology).
  • Context-Dependence: An algorithm built for a large city like NYC might not generalize to a small suburb without significant adjustment.
  • Strategic Behavior: In adversarial settings (like criminal justice), actors might try to "game" the algorithm, a phenomenon known as "data drift" or "adversarial response."
  • Bias: While the paper focuses on efficiency, the authors note that bias in underlying data can lead to biased algorithms, requiring careful design and evaluation.

Summary updated

Abstract

We calculate the social return on algorithmic interventions (specifically their Marginal Value of Public Funds) across multiple domains of interest to economists-regulation, criminal justice, medicine, and education.Though these algorithms are different, the results are similar and striking.Each one has an MVPF of infinity: not only does it produce large benefits, it provides a "free lunch."We do not take these numbers to mean these interventions ought to be necessarily scaled, but rather that much more R&D should be devoted to developing and carefully evaluating algorithmic solutions to policy problems.
BibTeX
@misc{ludwig2024unreasonable,
  author = {Jens Ludwig and Sendhil Mullainathan and Ashesh Rambachan},
  title = {{The Unreasonable Effectiveness of Algorithms}},
  year = {2024},
  doi = {10.2139/ssrn.4716690},
  howpublished = {SSRN},
}

Loading related content...

Figure