Sharp Optimality of Simple, Plug-in Estimation of the Fisher Information of a Smoothed Density

Subhodh KotekalICML 2025 · OpenReview

Introduction

In statistics and machine learning, estimating "functionals" of a probability density—such as entropy, mutual information, or Fisher information—from a finite set of data points is a fundamental task. Fisher information is notoriously difficult to estimate. In the standard "unsmoothed" case (estimating directly from the raw density), simple plug-in estimators—where one estimates the density first and then plugs it into the formula—are known to be suboptimal. To achieve accurate results, researchers typically have to rely on complex "debiasing" steps.

Recently, "Gaussian smoothing" (convolving the data with Gaussian noise) has been proposed to make these problems more tractable and to bypass the curse of dimensionality. However, the exact difficulty of estimating Fisher information after smoothing, and how the error scales with the variance of the smoothing noise ($t$), remained unclear.

This paper bridges that gap by establishing the sharp minimax rates (the theoretical "speed limit" of estimation) for this problem. Surprisingly, it proves that for Gaussian-smoothed densities, simple plug-in estimators are optimal, completely removing the need for complex debiasing. Furthermore, this result provides a solid theoretical foundation for estimating mutual information and entropy at the ideal parametric rate ($1/\sqrt{n}$).

Methodology: Minimax Theory and Noise Regimes

The author utilizes minimax theory to determine the fundamental difficulty of the estimation problem, focusing on a class of $\alpha$-Hölder densities that are compactly supported (e.g., existing only within $[-1, 1]$).

The paper proposes three distinct plug-in estimators tailored to three specific "noise regimes" based on the smoothing parameter $t$:

  1. Very high noise ($t \gtrsim 1$): Employs a modified plug-in strategy.
  2. High noise ($n^{-2/(2\alpha+1)} \lesssim t \lesssim 1$): Utilizes an unbiased estimator of the smoothed density.
  3. Low noise ($t \lesssim n^{-2/(2\alpha+1)}$): Relies on kernel density estimation of the underlying density.

To prove that these estimators achieve the best possible performance, the author employs a two-point construction for the lower bounds. This technique uses a "tunable parameter" approach to handle the specific challenges of smoothed densities, creating subtle perturbations of a base density to test the theoretical limits of any estimator.

Illustration of a central density perturbation ($f_2$) compared to the base density ($f_0$) used in the two-point construction for minimax lower bounds.
Illustration of a boundary density perturbation ($f_1$) compared to the base density ($f_0$), highlighting the specific challenges near the edges of the support.
(Note: This is a purely theoretical paper; no empirical datasets were used for training or evaluation. The results are derived mathematically for the general class of $\alpha$-Hölder probability densities.)

Key Insights and Results

Plug-in is Enough

The most surprising finding is that for any amount of smoothing ($t > 0$), simple plug-in estimators are optimal. The complex debiasing required for raw data becomes unnecessary once even a tiny bit of Gaussian noise is introduced.

The Sharp Minimax Rate

The paper establishes the exact minimax rate for estimating smoothed Fisher information ($I_t$). The error scales according to the following formula:

$$\text{Error} \approx \frac{1}{\sqrt{nt^2}} \wedge \frac{1}{\sqrt{nt^{3/4}}} \wedge \frac{n^{-\alpha/(2\alpha+1)}}{\sqrt{t}}$$

This reveals that the difficulty of the problem does not change linearly with $t$. Instead, it transitions through the three distinct mathematical regimes mentioned above, where the error scales differently (e.g., scaling with $t^{-2}$ versus $t^{-3/4}$).

The $t \to 0$ Blowup

As the smoothing parameter $t$ approaches zero, the estimation error "blows up." Fisher information is highly sensitive to the sharp edges (discontinuities) at the boundaries of the original data's compact support. While smoothing "smears" these edges, as the smoothing vanishes, the derivative of the density at these boundaries explodes. Qualitatively, Gaussian smoothing changes the estimation problem because it transforms the support of the density from bounded to unbounded.

Parametric Rates for Mutual Information and Entropy

By linking Fisher information to Mutual Information (MI) and Entropy via the I-MMSE and de Bruijn identities, the author demonstrates a powerful downstream application. We can estimate MI and Entropy at the "gold standard" parametric rate of $1/\sqrt{n}$ using simple plug-in estimators. Crucially, unlike Fisher information, the errors for MI and Entropy do not blow up as $t \to 0$.

Limitations and Future Work

While the paper provides a comprehensive theoretical framework, it notes a few areas for future exploration:

  • The "Slack" Gap: There is a minor mathematical gap in the lower bound proof for intermediate values of $t$ (between a small constant $c$ and a large constant $C$). The author conjectures this is a technical artifact of the proof rather than an inherent property of the problem.
  • Density Assumptions: The current analysis assumes the underlying density $f$ is bounded away from zero on its support. The behavior when the density can be arbitrarily close to zero remains an open question.
  • Dimensionality: The analysis is primarily presented for one-dimensional data. Because smoothing is frequently used to combat the curse of dimensionality in high-dimensional settings, deriving the specific "sharp" rates for dimensions $d > 1$ is a critical next step for future research.

Summary updated

Abstract

Given independent and identically distributed data from a compactly supported, $\alpha$-Hölder density $f$, we study estimation of the Fisher information of the Gaussian-smoothed density $f*\varphi_t$, where $\varphi_t$ is the density of $N(0, t)$. We derive the minimax rate including the sharp dependence on $t$ and show some simple, plug-in type estimators are optimal for $t > 0$, even though extra debiasing steps are widely employed in the literature to achieve the sharp rate in the unsmoothed ($t = 0$) case. Due to our result's sharp characterization of the scaling in $t$, plug-in estimators of the mutual information and entropy are shown to achieve the parametric rate by way of the I-MMSE and de Bruijn's identities.
BibTeX
@inproceedings{kotekal2025sharp,
  author = {Subhodh Kotekal},
  title = {{Sharp Optimality of Simple, Plug-in Estimation of the Fisher Information of a Smoothed Density}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=R6aLcBGBXf},
}
Figure