This 1922 paper by R.A. Fisher is arguably the "Big Bang" of modern statistical theory. Before this, statistics was a collection of ad-hoc methods and "common-sense" rules. Fisher provides the first rigorous mathematical framework for the field, defining what we are actually trying to do when we analyze data.
TL;DR
Fisher formalizes statistics as the "reduction of data" and introduces the Method of Maximum Likelihood. He establishes the three pillars of estimation—consistency, efficiency, and sufficiency—proving that we can mathematically determine which statistical methods extract the most information from a sample.
What Problem Does This Paper Address?
In the early 20th century, statisticians often confused the properties of a sample (the data you have) with the population (the hypothetical infinite group the data came from). Methods like the "Method of Moments" were popular but lacked a theoretical guarantee of quality.
Furthermore, the field was stuck in a debate over "inverse probability" (what we now call Bayesian inference). Critics argued that the Bayesian approach required arbitrary assumptions about "prior" knowledge (Bayes' Postulate). Fisher sought to create a theory of estimation that relied solely on the observed data, without needing these arbitrary priors.
Methods
Fisher introduces several concepts that are now standard in every statistics textbook:
- The Method of Maximum Likelihood (MLE): Instead of asking "What is the probability that these parameters are true?" (the Bayesian question), Fisher asks "Which parameter values make the observed data most likely?"
- Likelihood vs. Probability: He distinguishes the two, noting that while probability refers to the data, likelihood refers to our state of information about the parameters. Crucially, likelihood does not obey the laws of probability (e.g., it doesn't have to integrate to 1).
- The Three Criteria for Estimators:
- Consistency: As your sample size grows to infinity, your estimate should converge to the true population value.
- Efficiency: Among all consistent estimators, the "efficient" one is the one with the smallest possible variance (the least "noise").
- Sufficiency: A "sufficient statistic" is one that captures every single bit of relevant information in the data. Once you calculate a sufficient statistic, the rest of the raw data is useless for estimating that specific parameter.
Datasets
As a theoretical paper, Fisher primarily uses mathematical probability distributions (Normal, Poisson, and the Pearson system of curves) to test his ideas.
However, he does include a practical demonstration using a "dataset" of digits extracted from a 1764 table of Natural Logarithms (Vega). He uses these to simulate 25 samples of 100 observations to prove that for certain distributions (like a rectangular/uniform distribution), the standard arithmetic mean is a terrible (inefficient) way to estimate the center compared to using the extreme values of the sample.
Key Insights and Ideas
- Statistics as Data Reduction: Fisher argues the whole point of statistics is to replace a massive, unreadable pile of data with a few numbers (parameters) that describe the population exhaustively.
- The Failure of the Mean: One of the most "aha" moments is Fisher’s demonstration that the arithmetic mean—the most common tool in science—is not always the best tool. For some distributions, the mean is "entirely valueless" and provides no more information than a single random observation.
- Intrinsic Accuracy: Fisher introduces the idea that the "shape" of a distribution determines how much information it can possibly yield.
Key Results
- Superiority of MLE: Fisher provides a heuristic proof (later formalized by others) that the Method of Maximum Likelihood will always lead to a sufficient statistic if one exists.
- Quantifying Inefficiency: He shows that the popular "Method of Moments" is often only about 80% efficient for many common distributions. This means that using the Method of Moments is equivalent to throwing away 20% of your data.
- Information Theory Roots: By using the second derivative of the log-likelihood (the Hessian), Fisher provides the mathematical machinery to calculate the "weight" of information that each observation adds to an estimate.
Limitations and Future Work
Fisher is remarkably humble about his proofs, noting that he is "not satisfied as to the mathematical rigour" of his proof that MLE always leads to a sufficient statistic, inviting pure mathematicians to help formalize the work.
He also notes that his theories of "efficiency" are primarily "large-sample" theories (asymptotic). He acknowledges that for small samples, the distributions of these statistics might be discontinuous or complex, leaving a significant gap for future researchers to explore "small-sample" statistics—a field that would later be championed by his contemporary, "Student" (William Sealy Gosset).
Summary updated