Machine Learning Beyond Statistical Consistency for Hybrid AI

Jinsook Kim, Jinho KangarXiv 2026 · arXiv

TL;DR

This paper redefines machine learning as "successful computation," arguing that genuine learning requires more than just statistical convergence—it requires a formal, deductive proof of that convergence within the system itself. By extending Gödel’s Representability Theorem to empirical data, the authors provide a unified framework for "Hybrid AI" that bridges the gap between probabilistic prediction and logical verification.

What Problem Does This Paper Address?

Standard statistical learning theory (like the PAC learning framework) defines success through statistical consistency: the idea that as we see more data, our model’s estimate converges to the truth. However, this convergence usually relies on "global assumptions," most notably the i.i.d. assumption (that data points are independent and identically distributed).

The authors argue that these assumptions create a "circularity" problem. If a machine must assume the structure of the data to learn the data, is it actually learning, or just performing a pre-programmed calculation? Furthermore, modern AI is increasingly "hybrid"—performing both probabilistic tasks (like predicting the next word in a sentence) and deductive tasks (like writing code or proving theorems). Existing theories struggle to explain how these two fundamentally different modes of thought operate within a single formal system.

Methods

The authors build their framework on three pillars:

  1. Gödel’s Representability Theorem: Traditionally, this theorem states that any function a machine can compute is equivalent to a derivation in a formal system of arithmetic. The authors extend this to computable reals, allowing the system to handle probabilities.
  2. Empirical Definitions ($S_t$): They introduce a formal system, $S_t$, that treats observed data not as external input, but as "empirical definitions" (axioms) within the logic itself.
  3. The Two Conditions for Learning:
    • Truth Condition: The estimate must converge to the true probability (statistical consistency).
  4. Assurance Condition: The machine must be able to properly derive its own convergence within the formal system using only the record of data, not global assumptions.

A "proper derivation" is one that relies only on axioms and recorded data. If a machine requires a global assumption (like i.i.d.) to prove it is converging, the authors label this "mere computation" rather than "genuine learning."

Datasets

As a theoretical paper, it does not use standard empirical benchmarks like ImageNet or GLUE. Instead, it introduces a theoretical construct:

  • Ideal Corpus ($W^*$): Defined as the set of all words and sentences actually employed by a group of communicators up to a specific time. The authors use this to prove that N-gram models in Natural Language Processing (NLP) can meet the conditions for genuine learning because the "truth" is contained within the finite record of human speech.

Key Insights and Ideas

  • Epistemic Circularity: The paper’s most striking critique is directed at PAC learning. They prove that when the target is a true probability, the i.i.d. assumption is "epistemically circular"—it assumes the very structure the machine is trying to learn.
  • Learning vs. Lucky Guessing: A machine that converges to the truth but cannot prove why it is converging is merely a "lucky guesser." Genuine learning requires "Assurance"—the internal logical certainty of convergence.
  • The Hybrid AI Unified Field Theory: The authors show that proving, coding, and probabilistic learning are all just different types of derivations within the same formal system. Proving and coding use mathematical premises; learning uses empirical premises.
  • The "Clue" to Scaling Laws: The authors suggest that the success of Large Language Models (LLMs) despite the lack of i.i.d. data is because they operate in a domain (natural language) where precise learning is theoretically possible for an ideal machine, allowing real machines to approximate it effectively through massive scale.

Key Results

  • Theorem 2: Establishes that the Truth and Assurance conditions are both necessary and sufficient for a machine to learn a true probability.
  • Theorem 5: Formally demonstrates the epistemic circularity of the i.i.d. assumption in the PAC framework when learning true probabilities.
  • Theorem 6: Proves that the true probability of a sentence over a finite "Ideal Corpus" is indeed learnable under their new definition. This provides a theoretical "safe harbor" for NLP.

Limitations and Future Work

  • The Productivity Problem: The current framework handles sentences that have appeared in a corpus. However, human language is "productive"—we create and understand brand-new sentences. The authors acknowledge that learning the probability of a never-before-seen sentence remains an Open Problem.
  • Continuous vs. Discrete: There is a fundamental tension between the continuous math used in ML (like gradients) and the discrete logic of Turing machines. The authors admit that reconciling these "discrete epistemic bounds" with continuous models is a formidable challenge and the subject of their ongoing work.
  • Practical Approximation: While the paper focuses on "precise learning" by ideal machines, real-world AI uses "practical approximation." The authors argue that approximation is only possible where precise learning is possible, but they haven't yet provided a formal proof for the empirical success of scaling laws.

Summary updated

Abstract

We propose a new definition of machine learning, according to which a machine learns a function when it computes the function successfully, that is, when the value it returns for each input has a proper derivation in a formal system. We extend Gödel's Representability Theorem, which equates computation with derivation, to empirical targets in two ways: the theorem is extended to the computable reals, and the data are admitted into the system as empirical definitions that record what has been observed. We prove that a machine learns a true probability if and only if two conditions are met. The Truth Condition is that its estimate converges to the true probability, which is statistical consistency. The Assurance Condition is that the machine properly derives its own convergence inside the system. Learning thus lies beyond statistical consistency, for it requires a derivation of the convergence from the record. A global assumption about the data, such as the i.i.d. assumption, does not supply such a derivation, because it leads only to a circular one, as we show for PAC learning. On the other hand, the two conditions are met for the probability of a sentence over a finite ideal corpus. A real machine can approximate a true probability only if an ideal machine can learn it precisely, that is, only in a domain that meets the two conditions. Modern AI works in such a domain, and we offer this as a clue to its success. Modern AI is hybrid as well, in two senses: First, it performs two kinds of tasks, namely probabilistic ones such as predicting the next word and deductive ones such as proving and coding. Second, it combines two parts of different kinds, a statistical proposer and a deductive verifier, for proving and coding. Our framework explains both senses of hybridity in one formal system.
BibTeX
@misc{kim2026machine,
  author = {Jinsook Kim and Jinho Kang},
  title = {{Machine Learning Beyond Statistical Consistency for Hybrid AI}},
  year = {2026},
  eprint = {2611.01914},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01914},
}
Figure