Resolving the Reference Class Problem at Scale

Aaron Roth, Alexander TolbertPhilosophy 2025 · DOI

TL;DR

The "reference class problem"—the impossibility of uniquely determining an individual’s probability from finite data—is a classic hurdle in statistics. This paper argues that while this problem is unsolvable for a single person, it can be mitigated "at scale" by using model disagreements to constructively falsify and improve predictions across entire populations.

What Problem Does This Paper Address?

Imagine you want to predict the probability that "Alice" will be readmitted to a hospital. Should you group her with people of her age? Her zip code? Her specific medical history? Each "reference class" might yield a different probability, and with finite data, there is no mathematically "correct" choice. This is the individual reference class problem.

In modern machine learning, we face this at scale: predictive multiplicity. Two different models might both appear equally accurate and "well-justified" by the data, yet they give wildly different predictions for a large number of individuals. This creates an ethical and statistical paradox: if both models are "right" according to the data, how do we choose between them when their decisions (like insurance rates or medical care) change lives?

Methods

The authors bridge the gap between classical philosophy of science and modern algorithmic fairness. They utilize two primary frameworks:

  1. Multicalibration: A technique where a model is forced to be "consistent" (calibrated) not just on the whole population, but across a vast collection of overlapping subgroups (reference classes). If a model predicts a 20% risk for a group, roughly 20% of that group should actually experience the outcome.
  2. Model Reconciliation (Cross-Calibration): This is the paper's key "algorithmic" hero. If two models, $f_1$ and $f_2$, disagree significantly on a subset of people, that "disagreement region" is itself treated as a new reference class.
  3. The authors show that at least one of the two models must be statistically inconsistent on this disagreement region.
  4. By identifying this inconsistency, researchers can update the models to reduce their "Brier score" (a measure of prediction accuracy).

This process is iterative. You find where models fight, use that fight to prove one (or both) is wrong, and update them until they "march toward truth" and eventually agree.

Datasets

As a theoretical and foundational paper, it does not perform experiments on specific empirical datasets (like ImageNet or COMPAS). Instead, it provides a mathematical proof (Theorem 3.1) and a numerical simulation/example to demonstrate feasibility.

  • Theoretical Bound: The authors show that resolving disagreements to within a 5% margin ($\epsilon = 0.05$) with 95% confidence requires roughly 34,000 samples.
  • Scaling: Crucially, they note that the amount of data needed scales with the desired accuracy, not with the complexity or "richness" of the features (e.g., having thousands of data points per person doesn't make the reconciliation harder).

Key Insights and Ideas

  • Scale is a Feature, Not a Bug: In philosophy, the reference class problem is usually seen as a reason for despair. The authors flip this: when you make many predictions, the very arbitrariness that plagues individual cases becomes a statistical signal you can use to improve the whole system.
  • Constructive Falsification: Disagreement between two "equally good" models is actually an opportunity. It provides a roadmap to find where the evidence is lacking, allowing you to "falsify" the inferior parts of a model that would have otherwise gone unnoticed.
  • Independence from Feature Complexity: The reconciliation procedure works regardless of how many variables you have. You could have an "infinite" feature space (every conceivable fact about a person), and the statistical requirements for reconciliation remain the same.
  • The "Brier Score" Anchor: Because the Brier score is a "proper scoring rule," it acts as a gravitational pull toward the true underlying probabilities. Every time you fix a disagreement, the models' scores must improve or stay the same; they cannot get worse.

Key Results

  • Theorem 3.1 (Model Reconciliation): The authors formalize that for any two models $f_1$ and $f_2$, there exists an efficient algorithm that outputs a new pair of models ($f'_1, f'_2$) that:
  • Are more accurate (lower Brier score) than the originals.
  • "Almost agree almost everywhere" (the probability that they disagree by more than $\epsilon$ is itself less than $\epsilon$).
  • Convergence: The process is guaranteed to halt quickly. For a typical scenario, convergence can happen in about 800 rounds of updates, which takes seconds on modern hardware.
  • Resolution of the Paradox: They demonstrate that we can never be "stuck" with two equally well-justified but contradictory models at scale. We can always adjudicate between them using purely statistical means.

Limitations and Future Work

  • The Individual Still Suffers: The authors explicitly state that this does not solve the reference class problem for a single, unique individual. If Alice is truly one-of-a-kind in the data, her "true" probability remains elusive. Scale mitigates the problem for the population, but the "bite" of the problem remains for the individual.
  • Data Requirements: While 34,000 samples is small for "Big Data" companies, it may be a high bar for rare medical conditions or small-scale social studies.
  • Defining the Collection $\mathcal{S}$: The method assumes we can identify and compute expectations over the reference classes. If the "disagreement region" is so complex that it cannot be efficiently described or sampled, the reconciliation might struggle.
  • Future Direction: A natural next step is applying this "reconciliation" framework to real-world "algorithmic fairness" audits to see if it can resolve disputes between competing "fair" models in the wild.

Summary updated

Abstract

Abstract We draw a distinction between the traditional reference class problem, which describes an obstruction to estimating a single individual probability—which we rename the individual reference class problem —and what we call the reference class problem at scale , which can result when using tools from statistics and machine learning to systematically make predictions about many individual probabilities simultaneously. We argue that scale actually helps to mitigate the reference class problem, and purely statistical tools can be used to efficiently minimize the reference class problem at scale, even though they cannot be used to solve the individual reference class problem.
BibTeX
@inproceedings{roth2025resolving,
  author = {Aaron Roth and Alexander Tolbert},
  title = {{Resolving the Reference Class Problem at Scale}},
  booktitle = {Philosophy},
  year = {2025},
  doi = {10.1017/psa.2025.17},
}
Figure