Optimal Estimation of the Null Distribution in Large-Scale Inference

Subhodh Kotekal, Chao GaoIEEE Trans. Information Theory 2025 · OpenReview

TL;DR

This paper tackles the challenge of accurately estimating the "null distribution" in large-scale statistical inference, particularly when a significant fraction of data points might be non-null (outliers). The authors derive sharp, information-theoretic limits for estimating the location and scale of this null distribution, showing that Gaussian noise properties allow for better performance than previously thought, especially when the number of non-nulls is large.

What Problem Does This Paper Address?

In large-scale statistical inference, like analyzing thousands of hypothesis tests simultaneously, we often assume that the tests with truly null hypotheses follow a standard distribution (e.g., a standard normal distribution). However, as Efron highlighted, this assumption can be dangerous. In reality, a non-trivial fraction of these tests might be "non-null" (i.e., their underlying hypotheses are false), and these non-nulls can subtly distort the observed distribution of test statistics. This makes estimating the true null distribution (which is unknown) a critical problem. The paper specifically addresses the scenario where the number of non-nulls, denoted by k, can be a substantial fraction of the total number of tests, n. Existing robust statistics methods struggle with consistency in this "large k" regime, and the paper aims to establish the fundamental limits of what's possible and to develop optimal estimators.

Methods

The core of the paper's methodology relies on Fourier analysis and minimax estimation theory.

  • Fourier-based Estimators: The authors leverage the fact that the data is assumed to be Gaussian. They use the empirical characteristic function of the data to estimate the unknown parameters. This approach is particularly effective because it can "denoise" the signal by exploiting the known properties of Gaussian noise, allowing them to disentangle the null distribution from the non-null effects.
  • Minimax Rate Analysis: A significant portion of the paper is dedicated to establishing information-theoretic lower bounds on the estimation error. This involves constructing challenging statistical testing problems and using techniques like the x²-divergence and total variation distance to prove that no estimator can perform better than the derived rates.
  • Adaptive Estimation: The paper also develops adaptive estimators that can achieve the optimal rates without knowing the exact number of non-nulls (k) beforehand, using methods like Lepski's method.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and derives results for a general statistical model.

Key Insights and Ideas

  • Gaussianity is Key: The paper demonstrates that the Gaussian nature of the noise is crucial. It allows for significantly better estimation rates compared to non-Gaussian settings (like Huber's contamination model) when the number of non-nulls (k) is large. This is because the characteristic function of Gaussian noise is known exactly, enabling effective deconvolution.
  • Regime-Dependent Performance: The possibility of consistent estimation of the location parameter ($\theta$) critically depends on the relationship between n and k. Specifically, consistency is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$, meaning the number of "true nulls" must be sufficiently larger than $\sqrt{n}$.
  • Sharp Minimax Rates: The paper derives precise minimax rates for estimating both the location ($\theta$) and scale ($\sigma^2$) of the null distribution. These rates are shown to be optimal and exhibit interesting dependencies on k and n, including terms like $\frac{k^2(n-2k)^2}{n^3}$.
  • Fourier Methods Outperform: Estimators based on the empirical characteristic function are shown to be optimal, outperforming methods that don't fully exploit the Gaussian structure.

Key Results

  • Location Estimation: Consistent estimation of the location parameter $\theta$ is possible if and only if $\frac{n}{2} - k = \omega(\sqrt{n})$. The minimax rate for $\theta$ is derived, with different forms depending on the ratio of k to $\sqrt{n}$.
  • Scale Estimation: The scale parameter $\sigma^2$ can be estimated consistently in the entire regime $k < \frac{n}{2}$. The derived minimax rate for $\sigma^2$ is also sharp and shows a logarithmic improvement over existing results in similar settings.
  • Adaptation to k: Adaptive estimators are developed that achieve the optimal minimax rates for both $\theta$ and $\sigma^2$ without requiring prior knowledge of k.

Limitations and Future Work

  • Assumptions on Non-Nulls: While the paper aims to impose minimal assumptions on the non-null effects, some of the theoretical derivations (especially for lower bounds) might implicitly rely on certain properties of the non-null distribution (e.g., bounded moments or signal decay).
  • Computational Complexity: While Fourier-based methods are generally efficient, the adaptive estimators might involve computationally intensive steps like grid searches, although the paper suggests ways to mitigate this.
  • General Noise Distributions: The paper primarily focuses on Gaussian noise. While it briefly discusses extensions to other symmetric noise distributions, a full characterization of minimax rates for arbitrary noise distributions remains an open problem.
  • Unknown k: While adaptive estimators for k are provided, the practical implementation and performance of these adaptive methods in real-world scenarios could be further explored.

Summary updated

Abstract

The advent of large-scale inference has spurred reexamination of conventional statistical thinking. In a series of highly original articles, Efron persuasively illustrated the danger for downstream inference in assuming the veracity of a posited null distribution. In a Gaussian model for n many z-scores with at most $k \lt \frac {n}{2}$ nonnulls, Efron suggests estimating the parameters of an empirical null $N(\theta , \sigma ^{2})$ instead of assuming the theoretical null $N(0, 1)$ . Looking to the robust statistics literature by viewing the nonnulls as outliers is unsatisfactory as the question of optimal rates is still open; even consistency is not known in the regime $k \asymp n$ which is especially relevant to many large-scale inference applications. However, provably rate-optimal robust estimators have been developed in other models (e.g. Huber contamination) which appear quite close to Efron’s proposal. Notably, the impossibility of consistency when $k \asymp n$ in these other models may suggest the same major weakness afflicts Efron’s popularly adopted recommendation. A sound evaluation thus requires a complete understanding of information-theoretic limits. We characterize the regime of k for which consistent estimation is possible, notably without imposing any assumptions at all on the nonnull effects. Unlike in other robust models, it is shown consistent estimation of the location parameter is possible if and only if $\frac {n}{2} {-} k = \omega (\sqrt {n})$ , and of the scale parameter in the entire regime $k \lt \frac {n}{2}$ . Furthermore, we establish sharp minimax rates and show estimators based on the empirical characteristic function are optimal by exploiting the Gaussian character of the data.
BibTeX
@article{kotekal2025optimal,
  author = {Subhodh Kotekal and Chao Gao},
  title = {{Optimal Estimation of the Null Distribution in Large-Scale Inference}},
  journal = {IEEE Trans. Information Theory},
  year = {2025},
  url = {https://openreview.net/forum?id=T5gGWA8c7N},
}
Figure