Entropy Estimation via Normalizing Flow

Ziqiao Ao, Jinglai LiAAAI 2022 · DOI

Introduction

Entropy is a cornerstone metric in information theory, physics, and machine learning, quantifying the uncertainty or information content of a probability distribution. For continuous variables, estimating differential entropy from finite samples is a significant challenge, particularly when the underlying probability density function is unknown. While k-nearest neighbor (k-NN) estimators like the Kozachenko-Leonenko (KL) and Kraskov-Stögbauer-Grassberger (KSG) methods are widely used, they suffer from a severe "curse of dimensionality." As the number of dimensions ($d$) increases, the estimation bias grows exponentially, often rendering these methods unreliable beyond 15-20 dimensions.

Addressing the Boundary Effect

A primary source of bias in traditional k-NN estimators is the "boundary effect." Standard k-NN cells are often assumed to be fully contained within the support of the distribution. However, near the edges of the data's support, these cells inevitably extend into regions of zero probability, leading to systematic underestimation of the local density and, consequently, biased entropy estimates.

To mitigate this, the authors propose Truncated k-NN Estimators (tKL and tKSG). These methods explicitly account for the boundaries of the data support (typically assumed to be a unit cube $[0, 1]^d$). By truncating the k-NN cells at these boundaries, the estimator avoids including "empty space," significantly improving accuracy.

Illustration of k-NN cell truncation at the distribution support boundary.

Uniformizing Mapping via Normalizing Flows

The core innovation of this work is the Uniformizing Mapping (UM) framework. The authors leverage Normalizing Flows—specifically Masked Autoregressive Flows (MAF)—to learn a transformation that maps complex, high-dimensional data into a near-uniform distribution within a unit cube.

This approach is grounded in a key theoretical insight: the truncated KL estimator is proven to have zero bias when the data is perfectly uniform. By using a Normalizing Flow to "uniformize" the data, the complex problem of high-dimensional entropy estimation is reduced to estimating the entropy of a nearly uniform distribution, which is a much simpler task for the truncated k-NN methods. The final entropy of the original distribution is then recovered by adding a correction term derived from the Jacobian of the learned transformation.

Experimental Results and Scaling

The UM-based estimators were evaluated against standard benchmarks, including multivariate Gaussian and complex Rosenbrock distributions. A critical finding is the method's robustness to high dimensionality. In tests involving Gaussian distributions up to 40 dimensions, the proposed UM-tKL and UM-tKSG estimators maintained low Root-Mean-Square Error (RMSE), whereas the errors for traditional KL and KSG estimators increased dramatically.

RMSE of entropy estimators as a function of dimensionality $d$.

Beyond synthetic benchmarks, the method demonstrated practical utility in Optimal Experimental Design (OED) for a Lotka-Volterra predator-prey model. The UM-based approach successfully identified optimal sampling times that maximized information gain, outperforming both traditional k-NN methods and pure Normalizing Flow-based density estimation.

Conclusion and Future Directions

By combining the geometric flexibility of Normalizing Flows with the theoretical guarantees of truncated k-NN estimators, this work provides a powerful tool for high-dimensional entropy estimation. While the current framework assumes known support boundaries and requires data splitting to prevent overfitting, it opens new avenues for research in Bayesian experimental design and high-dimensional sensitivity analysis. Future work may focus on extending these techniques to distributions with unknown or infinite support.

Summary updated

BibTeX
@inproceedings{ao2022entropy,
  author = {Ziqiao Ao and Jinglai Li},
  title = {{Entropy Estimation via Normalizing Flow}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i9.21237},
}
Figure