Some Applications of the Cramer-Rao Inequality

J. L. HodgesProceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability 1951 · DOI

This classic 1951 paper by J. L. Hodges, Jr. and E. L. Lehmann is a foundational text in mathematical statistics. It demonstrates how the Cramér-Rao (CR) inequality—traditionally used to find the minimum variance of unbiased estimators—can be repurposed to solve deeper problems in decision theory, specifically proving the admissibility and minimaxity of common estimators.

TL;DR

Hodges and Lehmann provide a powerful method to prove that standard estimators (like the sample mean) are "optimal" in a very strict sense. By linking the Cramér-Rao lower bound to differential inequalities, they show these estimators are both minimax and admissible, even when compared against complex sequential sampling strategies.

What Problem Does This Paper Address?

In point estimation, researchers often look for minimax estimators—those that minimize the maximum possible risk (loss). While Abraham Wald’s "Bayes method" was the standard way to find these, it had a major flaw: it didn't always guarantee admissibility. An estimator is inadmissible if there exists another estimator that is at least as good everywhere and strictly better in at least one scenario.

Proving that an estimator is admissible is notoriously difficult. Furthermore, there was the question of whether sequential estimation (deciding when to stop sampling based on the data already seen) could outperform traditional fixed-sample size estimation. The authors sought a unified way to prove that standard fixed-size estimators are as good as it gets.

Methods

The authors leverage the Cramér-Rao Inequality, which provides a lower bound on the variance of an estimator based on its bias and the "Fisher Information" of the underlying distribution.

  1. Risk Bound via Bias: They define a function $C_b(\theta)$ that represents the minimum possible risk (Mean Squared Error) for any estimator with a specific bias function $b(\theta)$.
  2. Theorem 1 (The Core Tool): They prove that if an estimator's risk hits this CR lower bound, and if its bias function is the unique solution to a specific differential inequality, then the estimator is admissible.
  3. Sequential Extension: They apply a version of the CR inequality developed by Wolfowitz for sequential procedures. This allows them to compare fixed-size estimators against sequential ones that have the same expected number of observations.

Datasets

As a theoretical statistics paper, no empirical datasets were used. Instead, the authors rigorously tested their method on four fundamental probability distributions:

  • Normal (estimating the mean)
  • Poisson (estimating the mean)
  • Binomial (estimating the probability of success)
  • Chi-square (estimating the variance)

Key Insights and Ideas

  • Bias as a Constraint: The authors shifted the problem from searching over all possible estimators to searching over possible bias functions. If you can show that no other bias function can produce a lower risk bound than the one you have, you've proven admissibility.
  • The Chi-Square Surprise: They show that for the Chi-square distribution, the unique admissible minimax estimate for the parameter $\theta$ is $g(X) = X/(n + 2)$. Interestingly, this is a biased estimator. The standard unbiased estimator is actually neither minimax nor admissible in this context.
  • Fixed vs. Sequential: A major "aha" moment is the realization that for many common problems, sequential sampling offers no advantage over fixed-size sampling if your goal is to minimize maximum risk for a given average sample size. The simple sample mean is often already unbeatable.

Key Results

  • Normal Distribution: The sample mean $\bar{X}$ is proven to be the unique admissible minimax estimate for the mean $\theta$ under squared error loss.
  • Poisson Distribution: $\bar{X}$ is admissible and minimax when the loss is defined as squared error divided by the mean ($\theta$).
  • Binomial Distribution: The authors confirm that the estimator $\frac{X}{\sqrt{n}(\sqrt{n}+1)} + \frac{1}{2(\sqrt{n}+1)}$ is the admissible minimax estimate for the success probability under standard squared error loss.
  • Admissibility in Sequential Space: The paper establishes that these fixed-sample results hold firm even when the competitor is a sequential procedure, provided the sequential procedure is "bounded" (has a maximum possible number of observations).

Limitations and Future Work

  • Regularity Conditions: The method relies on the Cramér-Rao regularity conditions (e.g., the ability to differentiate under the integral sign). This means the technique cannot be applied to distributions with "sharp" boundaries that depend on the parameter, like the Uniform distribution $U(0, \theta)$.
  • Loss Function Sensitivity: The results are highly dependent on the choice of loss function. For example, the Poisson mean is minimax under one loss function but might have infinite minimax risk under another.
  • Unrestricted Sequential Procedures: While the authors prove results for bounded sequential procedures, they acknowledge that proving admissibility for unrestricted (potentially infinite) sequential procedures is more complex, though they provide a limiting argument to suggest their results still hold.

Summary updated

BibTeX
@inproceedings{hodges1951some,
  author = {J. L. Hodges},
  title = {{Some Applications of the Cramer-Rao Inequality}},
  booktitle = {Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability},
  year = {1951},
  doi = {10.1525/9780520411586-003},
}
Figure