The Theory is Predictive, but is it Complete? An Application to Human Perception of Randomness

Jon Kleinberg, Annie Liang, Sendhil Mullainathan

TL;DR

This paper introduces the concept of "theory completeness"—measuring not just if a theory is correct, but how much of the total predictable variation it actually captures. By using machine learning as an "optimal benchmark," the authors demonstrate that leading behavioral models of human randomness perception capture only about 15% of the predictable structure in the data, leaving significant room for theoretical improvement.

What Problem Does This Paper Address?

In the social sciences, a theory is often deemed successful if its predictions are statistically significant and better than random guessing. The authors argue this is a low bar. A theory might be "predictive" (correct in direction) but "incomplete" (missing most of what could be predicted).

The core challenge is that we usually don't know the "speed limit" of prediction for a given problem. If a model achieves 55% accuracy in predicting stock movements, is that a triumph or a failure? Without knowing the maximum achievable accuracy (the Bayes error rate), we can't judge how much room is left for better theories. This paper proposes using machine learning—specifically "atheoretical" models—to establish that missing benchmark.

Methods

The authors focus on a classic problem: human generation of random sequences. Humans are notoriously bad at being random; we tend to over-alternate (avoiding long runs of the same result) and misperceive the likelihood of streaks.

To measure completeness, they use three levels of modeling:

  1. Naive Baseline: Random guessing (e.g., a 50/50 chance for a coin flip).
  2. Existing Behavioral Theories: Models like Rabin (2002) and Rabin & Vayanos (2010) which formalize the "Gambler's Fallacy."
  3. The Benchmark (Table Lookup): Since the input space is small (e.g., sequences of 8 bits), they use a "brute force" table lookup. By observing thousands of human sequences, they calculate the exact empirical probability of what comes next for every possible prefix. This serves as a proxy for the best possible prediction.

They define Completeness as the ratio: $$\frac{\text{Theory Gain over Naive}}{\text{Benchmark Gain over Naive}}$$

Datasets

  • Primary Testbed: 21,975 human-generated strings of length 8 collected via Amazon Mechanical Turk.
  • Transfer Datasets: To test robustness, they used different alphabets (e.g., 'r' and '2' instead of 'H' and 'T'), different string lengths (up to 15), and an external dataset from Nickerson & Butler (2009).
  • Field Data:
    • Baseball Umpires: 1.5 million pitches where umpires call balls and strikes (testing the tendency to avoid calling too many strikes in a row).
  • Rock-Paper-Scissors: 29,864 strings of throws from a Facebook app, where players exhibit strategic dependencies.
Figure 1: Comparison of human-generated strings vs. true Bernoulli processes. Humans under-generate long runs and over-generate balanced strings.

Key Insights and Ideas

  • The 15% Ceiling: Across various experiments, existing behavioral models consistently explained only about 13–15% of the predictable variation. This suggests that while these theories are "correct," they are far from "complete."
  • Features vs. Combinations: The authors tested whether the gap was due to missing features or poor combination rules. They found that standard ML algorithms (Lasso, Decision Trees) using the same features as the behavioral models achieved 80-90% of the benchmark's performance.
  • The "Right" Features, the "Wrong" Math: This implies the research community already knows the "right" features (like run length and alternation rate), but the simple functional forms used in traditional theories are too restrictive to capture how those features actually interact in human behavior.
Figure 3: A simple two-split decision tree for predicting the next flip. Even this tiny ML model significantly outperforms complex behavioral theories.

Key Results

  • Predictive Error: In the "Continuation" task (predicting the 8th flip), the naive error was 0.250. Behavioral models improved this to ~0.249. The Table Lookup benchmark reached 0.243.
  • Stability: The completeness of the Rabin & Vayanos model remained remarkably stable across domains: ~15% for coin flips, ~17-21% for baseball umpires, and ~6-11% for Rock-Paper-Scissors.
  • ML as a Proxy: The results suggest that in complex domains where a "table lookup" isn't feasible, scalable ML algorithms can serve as highly effective benchmarks for testing the completeness of scientific theories.

Limitations and Future Work

  • Dimensionality: The "table lookup" approach only works for problems with a small number of features (low-dimensional input).
  • Interpretability: While ML shows that more can be predicted, it doesn't always explain why. The authors suggest the next step for theorists is to look at the gaps identified by ML to inspire more sophisticated, yet still interpretable, behavioral models.
  • Unmeasured Features: The authors acknowledge that adding new types of data (like the time taken to generate a flip) could raise the benchmark even higher, further reducing the perceived completeness of current theories.

Summary updated

Abstract

When we test a theory using data, it is common to focus on correctness: do the predictions of the theory match what we see in the data? But we also care about completeness: how much of the predictable variation in the data is captured by the theory? This question is difficult to answer, because in general we do not know how much "predictable variation" there is in the problem. In this paper, we consider approaches motivated by machine learning algorithms as a means of constructing a benchmark for the best attainable level of prediction. We illustrate our methods on the task of predicting human-generated random sequences. Relative to an atheoretical machine learning algorithm benchmark, we find that existing behavioral models explain roughly 15 percent of the predictable variation in this problem. This fraction is robust across several variations on the problem. We also consider a version of this approach for analyzing field data from domains in which human perception and generation of randomness has been used as a conceptual framework; these include sequential decision-making and repeated zero-sum games. In these domains, our framework for testing the completeness of theories provides a way of assessing their effectiveness over different contexts; we find that despite some differences, the existing theories are fairly stable across our field domains in their performance relative to the benchmark. Overall, our results indicate that (i) there is a significant amount of structure in this problem that existing models have yet to capture and (ii) there are rich domains in which machine learning may provide a viable approach to testing completeness.
BibTeX
@inproceedings{kleinberg2017theory,
  author = {Jon Kleinberg and Annie Liang and Sendhil Mullainathan},
  title = {{The Theory is Predictive, but is it Complete? An Application to Human Perception of Randomness}},
  booktitle = {EC},
  year = {2017},
  doi = {10.1145/3033274.3084094},
}
Figure