New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit

Aapo Hyvärinen

Introduction

In fields like Independent Component Analysis (ICA) and projection pursuit, the primary objective is often to discover "interesting" directions or hidden factors within complex data. In information theory, an "interesting" direction is typically synonymous with a "non-Gaussian" one. Differential entropy serves as the theoretically ideal measure for this task because entropy is maximized by the Gaussian distribution. Consequently, minimizing entropy allows researchers to isolate the most structured, non-Gaussian components of a dataset.

However, estimating differential entropy in practice is notoriously difficult. Traditional approaches generally force a difficult choice: either estimate the entire probability density function—which is computationally expensive and requires massive amounts of data—or rely on mathematical "shortcuts" like Gram-Charlier expansions.

The Flaw in Traditional Entropy Estimation

Historically, the shortcuts used to estimate entropy have relied heavily on higher-order cumulants, specifically kurtosis and skewness. The fundamental problem with these cumulant-based methods is that they act as "outlier magnets."

Kurtosis is essentially the expectation of $x^4$. Because the function $x^4$ grows so rapidly, any resulting estimate is overwhelmingly dominated by the tails of the distribution. A single extreme data point or a small amount of real-world noise can completely skew a kurtosis calculation, rendering the resulting entropy estimate highly unreliable for practical applications.

A Robust Maximum Entropy Method

To overcome the fragility of cumulant-based estimates, this paper proposes a paradigm shift using an approximative maximum entropy approach. Instead of attempting to guess the exact density of a variable, the methodology asks: What is the density that has the maximum possible entropy while still being consistent with a few simple measurements (expectations) taken from our data?

The author outlines several key technical steps to achieve this:

  1. Density Expansion: Deriving a first-order approximation of the maximum entropy density.
  2. Non-Quadratic Constraints: Instead of using rapidly growing polynomials (like $x^3$ or $x^4$), the author advocates for functions that grow slowly. Specifically, functions that do not grow faster than $x^2$ (such as $\log(\cosh(x))$ or $\exp(-x^2/2)$) are preferred. These non-quadratic functions focus the estimator's "attention" on the center of the distribution where the actual structure lies, making them far less sensitive to outliers in the tails.
  3. Orthonormalization: Employing a Gram-Schmidt-like process to ensure these new "measuring functions" are orthogonal to the Gaussian density and its first two moments (mean and variance).
  4. The Final Formula: This process yields a simple, elegant, and computationally efficient approximation: $$H(X) \approx H(\text{Gaussian}) - \sum k_i [E\{G_i(X)\} - E\{G_i(\nu)\}]^2$$ where $G_i$ represents the chosen robust measuring functions.

Experimental Validation and Key Results

Because the focus of the paper is on the mathematical validity of the entropy approximation itself, the author validated the new formulas using synthetic datasets with known "ground truth" entropy values. These included a family of Gaussian Mixture Densities (to simulate asymmetric densities with varying kurtosis) and a parameterized Symmetric Density Family (to test both sparse/high-kurtosis and sub-Gaussian/low-kurtosis distributions).

The results demonstrated a clear superiority of the proposed approximations (denoted as $H_a$ and $H_b$) over standard methods.

Comparison of negentropy approximations

As shown in the comparisons, the new approximations tracked the "true" negentropy curve much more closely than the cumulant-based method. Key findings include:

  • Superior Accuracy: For almost all tested non-Gaussian distributions, the proposed method provided a tighter, more reliable estimate of the information content.
  • Robustness to Sparsity: For "sparse" distributions featuring high positive kurtosis, traditional cumulant-based methods performed very poorly. In contrast, the new approximations remained highly accurate.
  • Computational Efficiency: Despite their robustness, the resulting formulas are just as easy to calculate as kurtosis, requiring only the average of a simple function over the data points.

Key Insights

Even if the proposed approximation isn't perfectly accurate in a global, absolute sense, it is highly "consistent" for the purpose of finding non-Gaussianity. Because the approximation reaches its maximum at the Gaussian distribution, it serves as a highly reliable and robust "index" for optimization algorithms in ICA and projection pursuit.

Limitations and Future Directions

While highly effective, the approach does have some limitations:

  • Near-Gaussian Assumption: The mathematical derivation assumes the underlying density is "not very far" from a Gaussian. While empirical results show it works well even for quite non-Gaussian data, the theoretical bounds are tightest near the Gaussian distribution.
  • Choice of Functions: The paper suggests a few optimal measuring functions, but notes that the absolute "best" function depends on the underlying data distribution, which is usually unknown in practice.
  • 1-D Focus: The current paper focuses strictly on one-dimensional entropy. While this is the foundational building block for ICA, extending these principles directly to high-dimensional joint entropy estimation remains an area for future work.

Summary updated

Abstract

We derive a first-order approximation of the density of maximum entropy for a continuous 1-D random variable, given a number of simple constraints. This results in a density expansion which is somewhat similar to the classical polynomial density expansions by Gram-Charlier and Edgeworth. Using this approximation of density, an approximation of 1-D differential entropy is derived. The approximation of entropy is both more exact and more ro(cid:173) bust against outliers than the classical approximation based on the polynomial density expansions, without being computationally more expensive. The approximation has applications, for example, in independent component analysis and projection pursuit.
BibTeX
@inproceedings{hyvarinen1997new,
  author = {Aapo Hyvärinen},
  title = {{New Approximations of Differential Entropy for Independent Component Analysis and Projection Pursuit}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1997},
}
Figure