Robustness of Standard Confidence Intervals for Location Parameters Under Departure from Normality

Sanjib Basu, Anirban DasguptaThe Annals of Statistics 1995 · DOI

Abstract

This study evaluates the conservatism of the standard Student’s $t$-interval $\bar{X} \pm t_{\alpha/2} s/\sqrt{n}$ for the unknown mean $\theta$ when the underlying distribution $F$ is non-normal. By analyzing the infimum of coverage probabilities across symmetric unimodal distributions, we identify a critical threshold for robustness. While high-confidence intervals (e.g., 95%, 99%) remain remarkably stable or conservative, intervals with low nominal coverage (where $t_{\alpha/2} < 1$) exhibit a collapse in reliability, with minimum coverage probabilities dropping to zero.

Introduction

A fundamental assumption in classical statistics is that data are sampled from a normal distribution. Under this assumption, the $t$-interval provides an exact confidence interval for the mean. While practitioners often rely on the Central Limit Theorem to justify the use of $t$-intervals for non-normal data, this asymptotic justification may fail for small to moderate sample sizes. This paper investigates the "worst-case" performance of the $t$-interval when the underlying distribution is symmetric and unimodal but potentially non-Gaussian.

Methodology

The authors determine the infimum coverage probability of the $t$-interval within the class of all symmetric unimodal distributions. The analysis utilizes the Khintchine representation, which allows any unimodal distribution to be expressed as a mixture of uniform distributions. This simplifies the optimization problem into finding the "worst" mixture. Additionally, independent embedding techniques and numerical minimization are employed to calculate exact coverage bounds for sample sizes ranging from $n=2$ to $n=10$.

The Robustness Threshold and the $t=1$ Discontinuity

A primary discovery is the existence of a sharp performance "cliff" related to the critical value $t$. For intervals where the critical value $t$ is significantly greater than 1, the $t$-interval is robust and often conservative. However, as the nominal confidence level decreases, there is a point $t_n$ below which the infimum coverage probability becomes zero. This means that for certain non-normal but symmetric and unimodal distributions, a low-confidence interval may have zero probability of containing the true mean.

Table 2: Robustness thresholds $t_n$ for various sample sizes. Below these values, infimum coverage drops to zero.

As shown in the table above, for a sample size of $n=10$, the interval maintains its integrity as long as the critical value $t$ is above 2.25.

Quantitative Performance

For standard scientific confidence levels, the $t$-interval performs admirably. In the family of symmetric unimodal distributions, the minimum actual coverage for a nominal 95% interval is very close to the target, even for small samples. This suggests that the $t$-test is "conservative" in high-stakes scenarios—it is safer than advertised.

Table 1: Infimum coverage probabilities for symmetric unimodal distributions across different sample sizes and nominal levels.

The data in Table 1 indicates that for $n=10$, a 95% nominal interval has a minimum actual coverage of 94.5%. Even at $n=2$, the 95% interval provides at least 92% coverage. This confirms that for high-confidence applications, the $t$-test is a robust choice. However, the table also illustrates how the infimum begins to diverge from the nominal value as the confidence level (and thus $t$) decreases.

Conclusion

The Student’s $t$-interval is surprisingly robust to departures from normality, provided the target confidence level is high. For 95% and 99% intervals, the actual coverage is often higher than or very close to the nominal value. The primary risk occurs at lower confidence levels or when interpreting $P$-values in large samples with slight contamination, where the test can become overly sensitive to tiny departures from normality. Researchers should remain cautious when using $t$-statistics for low-confidence estimations or in the presence of significant outliers.

Summary updated

Abstract

Let $X_i = \theta + \sigma Z_i$, where $Z_i$ are i.i.d. from a distribution $F$, and $-\infty < \theta < \infty$ and $\sigma > 0$ are unknown parameters. If $F$ is $N(0, 1)$, a standard confidence interval for the unknown mean $\theta$ is the $t$-interval $\bar{X} \pm t_{\alpha/2} s/\sqrt n$. The question of conservatism of this interval under nonnormality is considered by evaluating the infimum of its coverage probability when $F$ belongs to a suitably chosen class of distributions $\mathscr{F}$. Some rather surprising phenomena show up. For $\mathscr{F} = \{\text{all symmetric unimodal distributions}\}$ it is found that, for high nominal coverage intervals, the minimum coverage is attained at $U\lbrack -1, 1 \rbrack$ distribution, and the $t$-interval is quite conservative. However, for intervals with low or moderate nominal coverages $(t_{\alpha/2} < 1)$, it is proved that the infimum coverage is zero, thus indicating drastic sensitivity to nonnormality. This phenomenon carries over to more general families of distributions. Our results also relate to robustness of the $P$-value corresponding to the $t$-statistic when the underlying distribution is nonnormal.
BibTeX
@article{basu1995robustness,
  author = {Sanjib Basu and Anirban Dasgupta},
  title = {{Robustness of Standard Confidence Intervals for Location Parameters Under Departure from Normality}},
  journal = {The Annals of Statistics},
  year = {1995},
  doi = {10.1214/aos/1176324716},
}
Figure