Measuring the Completeness of Theories

Drew Fudenberg, Jon M. Kleinberg, Annie Liang, Sendhil Mullainathan

Abstract

Predictive accuracy is a standard but often misleading metric for evaluating scientific models, as it conflates a theory's structural shortcomings with the irreducible noise inherent in the data. This paper introduces "completeness," a formal metric that evaluates a model's performance relative to the maximum achievable predictive power for a given set of features. By utilizing a non-parametric machine learning benchmark—the "Table Lookup" algorithm—the authors establish a theoretical "speed limit" for accuracy. Applying this framework to classic economic problems, the study reveals whether predictive failures stem from inadequate model specifications or the fundamental unpredictability of the domain.

Introduction

In fields such as economics and psychology, the quality of a theory is typically judged by its predictive accuracy or error rates (e.g., Mean Squared Error). However, raw accuracy is notoriously difficult to interpret in isolation. A 55% accuracy rate may represent a breakthrough in financial forecasting but a failure in planetary physics. This discrepancy arises because different domains possess varying levels of intrinsic noise—variation that cannot be predicted using the available features.

When a model fails to predict perfectly, researchers face a critical dilemma: is the model’s mathematical structure flawed, or is the phenomenon inherently random given the measured variables? Without a way to separate these two sources of error, it is impossible to know whether to refine existing theories or pivot toward collecting entirely new types of data. This paper provides a quantitative framework to resolve this ambiguity.

Methodology: Defining Completeness

The authors propose measuring a model's completeness: the ratio of the improvement in prediction achieved by the model to the best possible improvement achievable by any model using the same features.

The Table Lookup Benchmark

To determine the "best achievable" prediction, the authors employ a non-parametric machine learning algorithm called Table Lookup. For every unique combination of features in the training data, Table Lookup calculates the average (or modal) outcome. Given a sufficiently large dataset, this brute-force approach captures the true underlying relationship between features and outcomes, effectively isolating the irreducible noise.

The Metric

Completeness is calculated as:

Completeness = (Naive Error - Model Error) / (Naive Error - Table Lookup Error)

In this equation, the "Naive Error" serves as a baseline (e.g., guessing at random). A completeness score of 1 implies the model has captured all available predictive power in the features, while a score of 0 implies it performs no better than the baseline.

Experimental Datasets

The framework was applied to three distinct behavioral prediction problems:

  1. Risk Preferences: 8,906 observations of "certainty equivalents" (the minimum guaranteed payout a subject accepts over a lottery) across 50 different lotteries.
  2. Initial Play in Games: 23,137 observations of how individuals play 486 different 3x3 matrix games upon first exposure.
  3. Human Randomness: A new dataset of 21,975 binary strings (length 8) where subjects were asked to mimic a fair coin toss, testing how humans perceive and generate randomness.

Key Insights and Results

The application of the completeness metric yielded several counter-intuitive findings that challenge traditional model evaluation:

  • High Error vs. High Completeness: In the risk preference task, Cumulative Prospect Theory (CPT) exhibited a high Mean Squared Error. Traditionally, this might suggest a poor model. However, the Table Lookup benchmark revealed that the best possible prediction also had a high error due to human inconsistency. CPT achieved 95% completeness, indicating that it captures nearly all predictable variation available in the features.
  • Low Completeness in Established Theories: In the human randomness task, existing behavioral models (e.g., Rabin 2002) achieved only 10-14% completeness. While these models improved slightly over random guessing, the Table Lookup showed a massive amount of predictable structure in human coin flips that current theories fail to account for.
  • Context-Dependent Performance: In game theory, the Poisson Cognitive Hierarchy Model (PCHM) was roughly 67% complete on general games. However, on a subset of games with heavily incentivized actions, its completeness jumped to 97%, suggesting the model is nearly optimal in specific contexts but lacks features to explain behavior in more complex scenarios.

Discussion and Limitations

The completeness metric serves as a strategic guide for future research. If a model is 95% complete, further refinement of its equations is unlikely to yield results; researchers should instead seek new features (e.g., response times or physiological data). Conversely, a model with 15% completeness suggests that the existing features contain significant untapped predictive power.

Despite its utility, the method has limitations. The Table Lookup benchmark is "data-hungry," requiring large samples relative to the number of feature combinations, making it less suitable for sparse or high-dimensional continuous spaces. Additionally, completeness is dataset-dependent; a model that is complete in one experimental setup may not generalize to others. Finally, while machine learning benchmarks excel at within-domain prediction, theory-driven models may still offer superior transferability to entirely new domains—a frontier for future research.

Summary updated

Abstract

Economic models are evaluated by testing the correctness of their predictions. We suggest an additional measure, “completeness”: the fraction of the predictable variation in the data that the model captures. We calculate the completeness of prominent models in three problems from experimental economics: assigning certainty equivalents to lotteries, predicting initial play in games, and predicting human generation of random sequences. The completeness measure reveals new insights about these models, including how much room there is for improving their predictions.
BibTeX
@article{fudenberg2021measuring,
  author = {Drew Fudenberg and Jon M. Kleinberg and Annie Liang and Sendhil Mullainathan},
  title = {{Measuring the Completeness of Theories}},
  journal = {Journal of Political Economy},
  year = {2021},
  doi = {10.1086/718371},
}
Figure