From Predictive Algorithms to Automatic Generation of Anomalies

Sendhil Mullainathan, Ashesh Rambachan

Abstract: How can we extract theoretical insights from machine learning algorithms? We take a familiar lesson: researchers often turn their intuitions into theoretical insights by constructing "anomalies" -- specific examples highlighting hypothesized flaws in a theory, such as the Allais paradox and the Kahneman-Tversky choice experiments for expected utility. We develop procedures that replace researchers' intuitions with predictive algorithms: given a predictive algorithm and a theory, our procedures automatically generate anomalies for that theory. We illustrate our procedures with a concrete application: generating anomalies for expected utility theory. Based on a neural network that accurately predicts lottery choices, our procedures recover known anomalies for expected utility theory and discover new ones absent from existing work. In incentivized experiments, subjects violate expected utility theory on these algorithmically generated anomalies at rates similar to the Allais paradox and common ratio effect.

Introduction

In economics, theoretical progress often hinges on the identification of "anomalies"—specific scenarios where a reigning theory's predictions fail. Historically, discovering these counter-examples, such as the Allais Paradox, required the creative intuition of researchers. While modern machine learning (ML) models can predict human behavior with high accuracy, they are often viewed as "black boxes" that lack theoretical depth. This research bridges that gap by transforming the creative act of anomaly generation into a formal optimization problem, using ML to systematically identify flaws in economic theories.

Methodology

The researchers treat an economic theory as an allowable function class—a set of rules restricting possible outcomes. To identify where these rules break, they propose two primary procedures:

  1. Adversarial Procedure: Framed as a max-min game, a "falsifier" algorithm searches for examples that maximize the theory's loss. It identifies data points where even the best-calibrated version of a theory cannot match the predictions of a high-accuracy ML model.
  2. Representational Morphing: This method exploits axioms like the independence axiom in Expected Utility Theory (EUT). The algorithm uses gradient descent to find scenarios that the theory considers identical but which the ML model predicts will cause a choice reversal in human subjects.

Experimental Validation

The approach was validated using three distinct data sources. First, simulated data from Prospect Theory was used to see if the algorithm could re-discover classic anomalies. Second, the authors utilized the Choices13K dataset, containing over 1 million decisions, to train a neural network that served as the "empirical intuition" for the search. Finally, the authors conducted new incentivized experiments with approximately 1,000 participants to test 60 of the newly discovered anomalies.

Key Results and Discovery

The procedures successfully re-discovered historical paradoxes, such as the Allais Paradox and the Common Ratio Effect, without prior knowledge of their existence.

The Allais Paradox, a classic anomaly for Expected Utility Theory recovered by the automated procedure.

Beyond recovering known effects, the algorithm generated hundreds of novel anomalies. For 2-payoff lotteries, it identified 881 verified anomalies, and for 3-payoff lotteries, it found 1,996. In real-world testing, human subjects violated EUT on these machine-generated menus at rates between 7.2% and 14.5%. These rates are statistically significant and comparable to the violation rates observed in the most famous paradoxes of behavioral economics.

Conclusion

This work demonstrates that machine learning can serve as a powerful source of intuition rather than just a replacement for theory. By treating economic anomalies as functional equivalents to "adversarial attacks" in computer vision, researchers can systematically uncover theoretical blind spots. While human researchers are still required to interpret and categorize these findings—leading to the identification of new phenomena like the "Strict Dominance Effect"—this automated pipeline significantly accelerates the discovery of theoretical boundary cases.

Summary updated

Abstract

How can we extract theoretical insights from machine learning algorithms? We take a familiar lesson: researchers often turn their intuitions into theoretical insights by constructing "anomalies" -- specific examples highlighting hypothesized flaws in a theory, such as the Allais paradox and the Kahneman-Tversky choice experiments for expected utility. We develop procedures that replace researchers' intuitions with predictive algorithms: given a predictive algorithm and a theory, our procedures automatically generate anomalies for that theory. We illustrate our procedures with a concrete application: generating anomalies for expected utility theory. Based on a neural network that accurately predicts lottery choices, our procedures recover known anomalies for expected utility theory and discover new ones absent from existing work. In incentivized experiments, subjects violate expected utility theory on these algorithmically generated anomalies at rates similar to the Allais paradox and common ratio effect.
BibTeX
@misc{mullainathan2024from,
  author = {Sendhil Mullainathan and Ashesh Rambachan},
  title = {{From Predictive Algorithms to Automatic Generation of Anomalies}},
  year = {2024},
  eprint = {2404.10111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.10111},
}

Loading related content...

Figure