The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis

Bradley EfronJournal of the American Statistical Association 1975 · DOI

TL;DR

Bradley Efron’s classic 1975 paper quantifies the "robustness tax" of choosing logistic regression over normal discriminant analysis. He proves that when data is truly normally distributed, logistic regression is significantly less efficient—often requiring 50% to 100% more data to achieve the same classification accuracy as its more specialized counterpart.

What Problem Does This Paper Address?

In the mid-70s, statisticians were increasingly choosing between two primary methods for binary classification: Normal Discriminant Analysis (NDA) and Logistic Regression (LR).

NDA (specifically Fisher’s linear discriminant) is the "gold standard" if you assume your data comes from two normal distributions with the same covariance. It uses the full information of the data's distribution to find the optimal boundary. LR, on the other hand, is more "robust"—it makes fewer assumptions about the underlying distribution of the features, focusing only on the conditional probability of the class given the features.

The central question Efron tackles is: If the normality assumptions actually hold, how much statistical efficiency do we sacrifice by using the more general logistic regression? In other words, what is the cost of being cautious?

Methods

Efron uses the framework of Asymptotic Relative Efficiency (ARE). He compares the two methods in a "standard situation" where a $p$-dimensional vector $\mathbf{x}$ comes from one of two normal populations with different means but an identical covariance matrix.

The key metric used is the Mahalanobis distance ($\Delta$), which measures how far apart the two population means are relative to their shared variance. A small $\Delta$ means the populations overlap heavily (hard classification), while a large $\Delta$ means they are well-separated (easy classification).

Efron derives a complex integral (Equation 1.10 in the paper) to calculate the ARE. This ratio represents the number of observations NDA needs divided by the number LR needs to achieve the same expected error rate as the sample size goes to infinity. He also breaks down the error into two components:

  1. Angle Error: How much the orientation of the estimated decision boundary deviates from the truth.
  2. Intercept Error: How much the boundary is shifted toward one population or the other.
This diagram illustrates the geometry of the decision boundary errors, showing the optimal boundary versus a non-optimum one defined by small discrepancies in angle ($d\alpha$) and intercept ($d\tau$).

Datasets

This is a theoretical paper and does not use empirical datasets. Instead, it relies on mathematical derivations based on the multivariate normal distribution model. The "results" are numerical evaluations of these theoretical formulas across different parameter ranges.

Key Insights and Ideas

  • The "Full" vs. "Conditional" Gap: NDA is a "full" maximum likelihood estimator (MLE) because it models the entire joint distribution of features and labels. LR is a "conditional" MLE because it only models the labels given the features. Information theory dictates that the full MLE will always be more efficient if its model is correct.
  • The Cost of Generality: The paper reveals that the "price" for LR's robustness is surprisingly high. For statistically "interesting" cases (where $\Delta$ is between 2.5 and 3.5), LR is often only 1/2 to 2/3 as effective as NDA.
  • Efficiency Drops as Separation Increases: Counter-intuitively, as the classification task becomes easier (larger $\Delta$), the relative efficiency of logistic regression actually decreases. When the populations are very distinct, NDA's knowledge of the distribution shape provides a massive advantage in pinning down the boundary precisely.

Key Results

The paper provides a definitive table (Table 1) showing ARE values for various distances ($\Delta$) and prior probabilities ($\pi_1$).

Table 1 shows the calculated relative efficiencies. For a typical case with equal priors ($\pi_1 = 0.5$) and a distance of $\Delta = 3$, the efficiency of logistic regression is only 0.641 compared to NDA.
  • At $\Delta = 0$: The methods are equally efficient (ARE = 1.000).
  • At $\Delta = 2.5$: LR is about 78% as efficient as NDA.
  • At $\Delta = 4.0$: LR's efficiency plummets to roughly 34%.
  • Robustness vs. Efficiency: While LR is less efficient under normality, Efron acknowledges it is valid under a much wider class of "exponential family" distributions. The statistician must decide if this added safety is worth the "moderately steep price" in sample size.

Limitations and Future Work

  • The "Home Court" Advantage: The entire comparison is conducted on NDA's "home court"—the assumption that the data is perfectly normal. The paper does not quantify how much better LR might perform if the data is non-normal (e.g., has heavy tails or outliers), which is the primary reason people use LR in practice.
  • Asymptotic Focus: The results are asymptotic ($n \to \infty$). The behavior in very small samples, where the differences might be even more pronounced or dominated by other factors, is not explored in detail here.
  • Nuisance Parameters: The paper notes that while LR is more robust to the shape of the distribution, it may be more sensitive to distorted sampling proportions (e.g., when one class is oversampled in a study), a topic Efron touches on in the final section.

Summary updated

BibTeX
@article{efron1975efficiency,
  author = {Bradley Efron},
  title = {{The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis}},
  journal = {Journal of the American Statistical Association},
  year = {1975},
  doi = {10.1080/01621459.1975.10480319},
}
Figure