Can Machines Learn the True Probabilities?

Jinsook KimICML 2024 · OpenReview

Introduction

When artificial intelligence systems operate under uncertainty, they are designed to make decisions that yield the best expected outcomes. These expectations rely heavily on probabilities. However, a deep philosophical and technical gap exists in probabilistic machine learning: are the probabilities generated by an AI merely subjective "guesses," or do they represent an objective "truth" about the environment?

This theoretical paper tackles a foundational question: Under what conditions can a machine actually learn the true objective probability of an event from data? By providing rigorous mathematical proofs, the author defines the fundamental limits of probabilistic machine learning, concluding that an AI can learn true objective probabilities if and only if those probabilities are directly observable.

The Problem: Subjective Guesses vs. Objective Reality

If a machine cannot learn the "true" probability of an event, its expectations are based on a flawed understanding of reality. This discrepancy can lead to sub-optimal or biased decision-making in critical, high-stakes fields such as causal inference or time-series forecasting. The paper seeks to bridge the gap between a model's subjective belief and objective truth, determining when a machine's probabilistic output can be trusted as a reflection of reality.

A Theoretical Framework for Learning

As a purely theoretical and mathematical work, this research does not rely on empirical datasets or benchmarks. Instead, it builds its arguments using mathematical logic and probability theory, applying them to any potential data-generating process (referred to as "Nature"). The methodology rests on several key pillars:

  • Computability Theory: Drawing on the foundational work of Turing and Church, the author defines a "machine" as an idealized device capable of effective calculation without practical limits on time or memory.
  • Forecasting Games: Learning is modeled as a game between "Nature" (which generates true facts) and a "Machine" (which generates subjective forecasts).
  • Success Criterion: A formal definition of learning is established. A machine is said to "learn" if it is "self-assured" to be correct most of the time and is actually correct with a probability of 1.
  • Calibration Theory: The proof utilizes Dawid’s notion of well-calibration—where a model’s predicted frequency matches the actual observed frequency—to link subjective forecasts with objective truth.

Key Insights and Core Theorems

The paper treats machine learning not merely as a technical optimization task, but as an epistemic act—the acquisition of knowledge that must truly represent the world.

Nature's Perversity and the Illusion of Learning

The author introduces the concept of "Nature's perversity." If Nature deviates from a machine's probabilistic forecast infinitely often, the machine can never truly learn the probability. Furthermore, a machine might "win" a forecasting game by making a correct prediction through luck or a specific strategy, but winning does not equate to having learned the underlying probability distribution.

The Observability Requirement

The central breakthrough of the paper is encapsulated in Theorem 4.36, which establishes that learning is equivalent to direct observability. If a probability cannot be reduced to an empirical frequency that the machine can calculate, it is fundamentally unlearnable.

This is supported by Theorem 4.32, which proves that a machine can only learn the true probability when it is "self-assured" of a "stopping time"—a point after which Nature no longer deviates from the machine's forecast. Consequently, if a machine does learn the true objective probability, its forecasts are "truly guaranteed" to be well-calibrated (Theorem 4.6).

Relaxing Assumptions

In practical terms, the paper suggests that while machines can relax the "independence" part of the standard i.i.d. (independent and identically distributed) assumption, they cannot relax the "identical distribution" requirement (such as stationarity or ergodicity) if their goal is to learn the objective truth.

Limitations and Future Directions

The proofs in this paper assume an "idealized" machine with limitless time and storage. Real-world machines, constrained by practical computational limits, face even stricter impossibilities. Additionally, the paper operates on the frequentist-leaning assumption that a "true objective probability" actually exists; if the world is fundamentally non-probabilistic, the premise shifts entirely.

Looking forward, the author suggests that future research should focus on the computational complexity of calculating these probabilities once they are deemed observable. There is also a rich opportunity to investigate how AI can adapt to "perverse" environments where the underlying rules of the forecasting game change over time.

Summary updated

Abstract

When there exists uncertainty, AI machines are designed to make decisions so as to reach the best expected outcomes. Expectations are based on true facts about the objective environment the machines interact with, and those facts can be encoded into AI models in the form of true objective probability functions. Accordingly, AI models involve probabilistic machine learning in which the probabilities should be objectively interpreted. We prove under some basic assumptions when machines can learn the true objective probabilities, if any, and when machines cannot learn them.
BibTeX
@inproceedings{kim2024can,
  author = {Jinsook Kim},
  title = {{Can Machines Learn the True Probabilities?}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=TzqmqZS0nj},
}
Figure