On the mathematical foundations of theoretical statistics

R. A. FisherPhilosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 1922 · DOI

This 1922 paper by R.A. Fisher is arguably the "Big Bang" of modern statistical theory. Before this, statistics was a collection of ad-hoc methods and "common-sense" rules. Fisher provides the first rigorous mathematical framework for the field, defining what we are actually trying to do when we analyze data.

TL;DR

Fisher formalizes statistics as the "reduction of data" and introduces the Method of Maximum Likelihood. He establishes the three pillars of estimation—consistency, efficiency, and sufficiency—proving that we can mathematically determine which statistical methods extract the most information from a sample.

What Problem Does This Paper Address?

In the early 20th century, statisticians often confused the properties of a sample (the data you have) with the population (the hypothetical infinite group the data came from). Methods like the "Method of Moments" were popular but lacked a theoretical guarantee of quality.

Furthermore, the field was stuck in a debate over "inverse probability" (what we now call Bayesian inference). Critics argued that the Bayesian approach required arbitrary assumptions about "prior" knowledge (Bayes' Postulate). Fisher sought to create a theory of estimation that relied solely on the observed data, without needing these arbitrary priors.

Methods

Fisher introduces several concepts that are now standard in every statistics textbook:

  • The Method of Maximum Likelihood (MLE): Instead of asking "What is the probability that these parameters are true?" (the Bayesian question), Fisher asks "Which parameter values make the observed data most likely?"
  • Likelihood vs. Probability: He distinguishes the two, noting that while probability refers to the data, likelihood refers to our state of information about the parameters. Crucially, likelihood does not obey the laws of probability (e.g., it doesn't have to integrate to 1).
  • The Three Criteria for Estimators:
    1. Consistency: As your sample size grows to infinity, your estimate should converge to the true population value.
  • Efficiency: Among all consistent estimators, the "efficient" one is the one with the smallest possible variance (the least "noise").
  • Sufficiency: A "sufficient statistic" is one that captures every single bit of relevant information in the data. Once you calculate a sufficient statistic, the rest of the raw data is useless for estimating that specific parameter.

Datasets

As a theoretical paper, Fisher primarily uses mathematical probability distributions (Normal, Poisson, and the Pearson system of curves) to test his ideas.

However, he does include a practical demonstration using a "dataset" of digits extracted from a 1764 table of Natural Logarithms (Vega). He uses these to simulate 25 samples of 100 observations to prove that for certain distributions (like a rectangular/uniform distribution), the standard arithmetic mean is a terrible (inefficient) way to estimate the center compared to using the extreme values of the sample.

Key Insights and Ideas

  • Statistics as Data Reduction: Fisher argues the whole point of statistics is to replace a massive, unreadable pile of data with a few numbers (parameters) that describe the population exhaustively.
  • The Failure of the Mean: One of the most "aha" moments is Fisher’s demonstration that the arithmetic mean—the most common tool in science—is not always the best tool. For some distributions, the mean is "entirely valueless" and provides no more information than a single random observation.
  • Intrinsic Accuracy: Fisher introduces the idea that the "shape" of a distribution determines how much information it can possibly yield.
Symmetrical error curves of equal intrinsic accuracy. Curve A is a Cauchy-like distribution (Type VII), while Curve B is a Normal distribution. Fisher uses this to show that distributions with very different shapes can still offer the same 'intrinsic accuracy' for locating the center of the distribution.

Key Results

  • Superiority of MLE: Fisher provides a heuristic proof (later formalized by others) that the Method of Maximum Likelihood will always lead to a sufficient statistic if one exists.
  • Quantifying Inefficiency: He shows that the popular "Method of Moments" is often only about 80% efficient for many common distributions. This means that using the Method of Moments is equivalent to throwing away 20% of your data.
  • Information Theory Roots: By using the second derivative of the log-likelihood (the Hessian), Fisher provides the mathematical machinery to calculate the "weight" of information that each observation adds to an estimate.

Limitations and Future Work

Fisher is remarkably humble about his proofs, noting that he is "not satisfied as to the mathematical rigour" of his proof that MLE always leads to a sufficient statistic, inviting pure mathematicians to help formalize the work.

He also notes that his theories of "efficiency" are primarily "large-sample" theories (asymptotic). He acknowledges that for small samples, the distributions of these statistics might be discontinuous or complex, leaving a significant gap for future researchers to explore "small-sample" statistics—a field that would later be championed by his contemporary, "Student" (William Sealy Gosset).

Summary updated

Abstract

Abstract Several reasons have contributed to the prolonged neglect into which the study of statistics, in its theoretical aspects, has fallen. In spite of the immense amount of fruitful labour which has been expended in its practical applications, the basic principles of this organ of science are still in a state of obscurity, and it cannot be denied that, during the recent rapid development of practical methods, fundamental problems have been ignored and fundamental paradoxes left unresolved. This anomalous state of statistical science is strikingly exemplified by a recent paper entitled "The Fundamental Problem of Practical Statistics," in which one of the most eminent of modern statisticians presents what purports to be a general proof of BAYES' postulate, a proof which, in the opinion of a second statistician of equal eminence, "seems to rest upon a very peculiar -- not to say hardly supposable -- relation."
BibTeX
@article{fisher1922mathematical,
  author = {R. A. Fisher},
  title = {{On the mathematical foundations of theoretical statistics}},
  journal = {Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character},
  year = {1922},
  doi = {10.1098/rsta.1922.0009},
}
Maximum Likelihood Estimation of Functionals of Discrete DistributionsarXiv · 2014
Some Applications of the Cramer-Rao InequalityProceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability · 1951
Maximum likelihood characterization of distributionsBernoulli · 2014
Outline of a Theory of Statistical Estimation Based on the Classical Theory of ProbabilityPhilosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences · 1937
Differential Geometry of Curved Exponential Families-Curvatures and Information LossThe Annals of Statistics · 1982
Theoretical Foundations of Equitability and the Maximal Information CoefficientarXiv · 2014
Figure