Abstract Every philosophy has holes, and it is the responsibility of proponents of a philosophy to point out these problems. Here are a few holes in Bayesian data analysis: (1) the usual rules of conditional probability fail in the quantum realm, (2) flat or weak priors lead to terrible inferences about things we care about, (3) subjective priors are incoherent, (4) Bayesian decision picks the wrong model, (5) Bayes factors fail in the presence of flat or weak priors, (6) for Cantorian reasons we need to check our models, but this destroys the coherence of Bayesian inference. Some of the problems of Bayesian statistics arise from people trying to do things they should not be trying to do, but other holes are not so easily patched. In particular, it may be a good idea to avoid flat, weak, or conventional priors, but such advice, if followed, would go against the vast majority of Bayesian practice and requires us to confront the fundamental incoherence of Bayesian inference. This does not mean that we think Bayesian inference is a bad idea, but it does mean that there is a tension between Bayesian logic and Bayesian workflow which we believe can only be resolved by considering Bayesian logic as a tool, a way of revealing inevitable misfits and incoherences in our model assumptions, rather than as an end in itself.
Introduction
In the landscape of modern statistics, Bayesianism is often championed as the "gold standard" of rationality. Its appeal lies in its perceived logical coherence—the idea that by following the laws of probability, one can arrive at a perfectly consistent system for updating beliefs. However, this idealized view often clashes with the reality of scientific practice. This paper identifies six fundamental "holes" in Bayesian statistics where the theory either fails to apply or leads to internally inconsistent results. Rather than dismissing Bayesian inference, the goal is to reframe it: moving from a focus on "Bayesian logic" as an unassailable philosophy to a "Bayesian workflow" that uses these very contradictions as diagnostic tools.
The Quantum Limitation
The first hole is found at the most fundamental level of physics. Standard Bayesian inference assumes that a joint probability distribution exists for all variables under consideration. However, quantum mechanics—specifically the two-slit experiment—demonstrates that this is physically false. In the quantum realm, the act of measuring one property can fundamentally alter the distribution of another. Superposition is not merely a state of "probabilistic uncertainty" that can be resolved with a standard joint distribution; it represents a breakdown of the classical rules of conditional probability upon which Bayesianism is built.
The Trap of Weak Priors
A common practice in Bayesian analysis is the use of "flat" or "weakly informative" priors, intended to let the "data speak for itself." Paradoxically, these priors often lead to inappropriately strong and overconfident conclusions.
Consider an estimate that is exactly one standard error away from zero. In a frequentist framework, this is clearly non-significant noise. However, under a flat prior, this results in a posterior probability of 84% that the effect is positive. In a real-world decision-making context, few researchers would actually bet 5-to-1 on an effect supported by such flimsy evidence. This discrepancy reveals that "non-informative" priors are rarely truly neutral; they often bake in assumptions that lead to over-inference from noisy data.
The Myth of Subjective Coherence
Bayesian theory often relies on the idea of a "subjective prior"—a pre-existing belief state that a researcher can perfectly define before seeing any data. In practice, this is a myth. Modeling is an inherently iterative process. Researchers often only realize their priors were unrealistic after seeing the posterior results. This creates a fundamental incoherence: if we change our priors because the posterior looks "wrong," we are no longer following the strict logical rules of Bayesian updating. This tension suggests that subjectivity is not a starting point, but a moving target.
Cantorian Model Expansion
Drawing on an analogy to Georg Cantor’s diagonal argument—which proves that you cannot list all real numbers—the authors argue that it is impossible to construct a single "super-model" that accounts for all future data surprises. Scientific progress requires "Cantorian updating": the process of expanding a model when it fails to capture new phenomena.
While this expansion is necessary for science, it is technically "illegal" within the strict confines of Bayesian logic. Checking a model against data and then modifying it destroys the internal logical coherence of the original probability space. Yet, without this process, statistics would be a closed loop, unable to learn from its own failures.
The Fragility of Bayes Factors
Bayes factors are frequently used for model comparison, yet they are notoriously sensitive to the exact specifications of a prior—even details that have no impact on the actual parameter estimates. This sensitivity makes them a fragile tool for scientific discovery. As an alternative, the authors advocate for "model stacking," where multiple models are averaged based on their predictive performance. Stacking offers a more robust path forward, prioritizing predictive utility over the "all-or-nothing" logic of Bayes factors.
Conclusion: From Logic to Workflow
The "holes" identified here do not suggest that Bayesian inference is a bad idea. Instead, they suggest that the value of Bayesianism lies not in its perfection as a logical system, but in its power as a diagnostic tool. The "Bayesian workflow"—the cycle of building, checking, failing, and expanding models—is where the real work of science happens. By accepting that statistical coherence is a goal we approach but never fully reach, we can use Bayesian logic to reveal the inevitable misfits between our models and the complex reality they attempt to describe.
Summary updated