From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian

Ya’acov RitovarXiv 2025 · DOI · arXiv

TL;DR

In the era of ultra-high-dimensional "Big Data," the classical Bayesian ideal—where a prior represents a researcher's honest, subjective beliefs—is becoming mathematically and practically unfeasible. Ya’acov Ritov argues that in high dimensions, "natural" priors fail in unpredictable ways, and the Bayesian procedures that actually work are often just frequentist estimators in a "Bayesian" costume.

What Problem Does This Paper Address?

For decades, the statistical community has been divided between Frequentists and Bayesians. The "Subjective Bayesian" (S-Bayesian) paradigm relies on the idea that we can encapsulate our pre-existing knowledge into a prior distribution.

However, modern statistics often deals with "ultra-high dimensional" models where the number of parameters ($p$) is much larger than the number of observations ($n$). Ritov tackles a fundamental question: Can a human (or even a computer) honestly maintain a "worldview" or a set of beliefs about thousands or millions of parameters simultaneously? He argues that as dimensionality explodes, the mathematical foundations that once supported subjective Bayesianism (like the seminal works of Wald and Savage) lose their relevance, leading to estimators that are either biased or logically inconsistent with the very idea of a "prior."

Methods

This is a theoretical paper that uses mathematical proofs and "toy models" to demonstrate the breakdown of Bayesian logic in high dimensions. Ritov distinguishes between three types of Bayesians:

  1. S-Bayesian (Subjective): Follows the paradigm that the prior represents true beliefs.
  2. F-Bayesian (Formal/Technical): Uses priors as a mathematical convenience to get robust estimators (e.g., using a Laplace prior to mimic the Lasso).
  3. M-Bayesian (Model-based): The prior has a frequentist interpretation (e.g., the physical properties of a machine).

The paper analyzes several specific scenarios:

  • Neyman-Scott models: Examining how hospital-specific effects behave when data is anonymized.
  • White Noise models: Estimating functionals of a signal.
  • Plug-in Estimation: Showing why Bayesian estimators struggle to reach optimal convergence rates for specific functionals because they are "locked" into a single prior.

Datasets

Not discussed in the paper. As a theoretical work, it relies on mathematical constructions and simulated scenarios (like the "White Noise" model) rather than empirical datasets.

Key Insights and Ideas

  • High-Dimensional Space is "Empty": In 100+ dimensions, intuition fails. For example, 99% of the volume of a high-dimensional ball is located in a tiny shell near the surface. This means two priors that look "similar" to a human can actually be "asymptotically orthogonal"—meaning they place their probability mass on completely different, non-overlapping parts of the space.
  • The "Plug-in" Trap: A subjective Bayesian must use the same prior for the whole model. However, frequentists know that to estimate different parts of a model efficiently (e.g., the mean vs. the derivative of a density), you often need different "tuning." Because the Bayesian is stuck with one prior, their "plug-in" estimators for specific functionals are often biased and cannot reach the optimal (minimax) convergence rates.
  • Priors that Depend on $n$: To make Bayesian methods work in high dimensions (like the Lasso), the "prior" often has to be tuned based on the sample size ($n$). This is a philosophical "no-go" for subjective Bayesians: a prior, by definition, should represent what you believe before you see the data (and thus before you know the sample size).
  • Consistency is a Weak Guarantee: Ritov proves that while Bayesian procedures are "consistent" under their own priors, this is a very weak statement in high dimensions. A prior can be "consistent" while being totally wrong about the actual underlying truth.

Key Results

  • Theorem 3.3.1 (Bias): Ritov provides a firm proof that Bayes estimators are inherently biased unless the estimator is perfectly accurate with probability 1. In high dimensions, this bias accumulates, making Bayesian "plug-in" estimators perform worse than frequentist alternatives.
  • The Failure of Exchangeability: The paper shows that assuming parameters are "exchangeable" (a common subjective shortcut) implies very strong, often false, claims about the means of subsets of those parameters.
  • Frequentist Mimicry: The paper demonstrates that "successful" high-dimensional Bayesian procedures (like those used in "Big Data" regression) are usually just frequentist tools (like the Lasso) re-interpreted. The priors used aren't "beliefs"—they are chosen specifically because they produce frequentist-friendly results.

Limitations and Future Work

The author acknowledges that Bayesian techniques remain incredibly useful for computation and for generating new types of estimators. The critique is specifically aimed at the subjective philosophy.

A major limitation noted is that the paper focuses on models that can be analyzed "transparently." Ritov suggests that if subjective Bayesianism fails in these relatively simple high-dimensional cases, it is even less likely to be valid in "black box" scenarios like deep neural networks. Future work in the field may need to move toward better justifying "Formal" priors that don't pretend to be subjective beliefs but are mathematically rigorous for high-dimensional inference.

Summary updated

Abstract

We argue that the Bayesian paradigm, of a prior which represents the beliefs of the statistician before observing the data, is not feasible in ultra-high-dimensional models. We claim that natural priors that represent the a priori beliefs fail in unpredictable ways under values of the parameters that cannot be honestly ignored. We do not claim that the frequentist estimators we present cannot be mimicked by Bayesian procedures, but that these Bayesian procedures do not represent beliefs. They were created with the frequentist analysis in mind, and in most cases, they cannot represent a consistent set of beliefs about the parameters (for example, since they depend on the loss function, the particular functional of interest, and not only on the a priori knowledge, different priors should be used for different analyses of the same data set). In a way, these are frequentist procedures using a Bayesian technique. The paper presents different examples where the subjective point of view fails. It is argued that the arguments based on Wald's and Savage's seminal works are not relevant to the validity of the subjective Bayesian paradigm. The discussion tries to deal with the fundamentals, but the argument is based on a firm mathematical proofs.
BibTeX
@misc{ritov2025from,
  author = {Ya’acov Ritov},
  title = {{From Thomas Bayes to Big Data: On the feasibility of being a subjective Bayesian}},
  year = {2025},
  eprint = {2508.01642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.01642},
  howpublished = {arXiv},
}
Figure