Optimum Fisherian Information for Multivariate Distributions

Tamer BaşarThe Annals of Statistics 1977 · DOI

Abstract

It is shown that within the class of all multivariate distributions depending on a location parameter (and satisfying certain smoothness conditions) and with a weighted norm constraint on the covariance matrix, the one with minimum Fisherian information is the Gaussian distribution. This result is then used in obtaining a tight upper bound on the error of estimating an unknown random vector observed in additive Gaussian noise under quadratic loss.

Introduction

In the fields of statistics and information theory, Fisher information serves as a fundamental metric for the amount of information an observable random variable carries about an unknown parameter. A well-established result in univariate statistics is that for a fixed variance, the Gaussian distribution minimizes Fisher information, effectively representing the "least informative" or "hardest to estimate" distribution.

Tamer Başar's work addresses the non-trivial challenge of generalizing this principle to multivariate ($n$-dimensional) distributions. Specifically, the paper investigates distributions where the parameter of interest is a location parameter—a shift in the mean—and the covariance matrix is subject to a weighted norm constraint. Identifying the least informative distribution under these conditions is crucial for establishing performance bounds in engineering applications, such as signal processing in noisy environments.

Methodology

The paper utilizes a rigorous mathematical framework grounded in functional analysis and variational methods. The proof follows several key steps:

  1. Defining the Constraint: Rather than a simple variance constraint, the author introduces a weighted trace constraint on the covariance matrix: $\text{Tr}[D \text{Cov}(X)] = k^2$, where $D$ is a positive definite matrix.
  2. Objective Function: The goal is to minimize a weighted trace of the Fisher information matrix, $\text{Tr}[C I_p]$, where $C$ is another positive definite matrix.
  3. Inner Product Spaces: The author defines a specific Hilbert space ($L_2$ space) of random vectors and an inner product related to the score function (the gradient of the log-density).
  4. Cauchy-Schwarz Inequality: By applying the Cauchy-Schwarz inequality within this inner product space, the author derives a lower bound for the Fisher information and demonstrates that this bound is uniquely attained by the Gaussian distribution.

Theoretical Insights

The core contribution of this research is the formalization of the Gaussian distribution as an extremal distribution. It is not merely a convenient model but is mathematically the "least favorable" distribution when attempting to extract information about a location parameter under power constraints.

Furthermore, the paper reveals that the optimal covariance structure of this least informative Gaussian is determined by the interplay between the weight matrix of the constraint ($D$) and the weight matrix of the information objective ($C$). This insight has significant implications for robust estimation: if an estimator is designed to perform well against Gaussian noise, it effectively provides a "minimax" solution—a guaranteed performance bound regardless of the actual distribution, provided the covariance constraints are met.

Key Results

The primary findings are summarized in two main points:

  • Theorem 1: The paper formally proves that the minimum of the weighted Fisher information is attained uniquely by a Gaussian distribution. It provides an explicit formula for this minimum value: $J^* = [\text{Tr}(F)]^2 / k^2$, where $F$ is a matrix derived from the square roots of $C$ and $D$.
  • Estimation Error Bound: In the context of estimating a signal $Y$ from $X = Y + W$ (where $W$ is Gaussian noise), the paper proves that the quadratic estimation error is maximized when the signal $Y$ itself is Gaussian. This establishes a tight, closed-form upper bound for the expected error in multivariate signal estimation.

Limitations and Future Work

While foundational, the results are subject to certain conditions:

  • Regularity Conditions: The proof assumes the density is continuously differentiable with specific tail behavior, excluding "rough" or heavy-tailed distributions.
  • Parameter Scope: The result is specific to location parameters and may not extend to scale parameters or other statistical manifolds.
  • Loss Function: The application focuses on quadratic loss; different loss functions might yield different extremal distributions.

Future research may explore minimum information proofs for alternative constraints or non-Gaussian noise environments.

Summary updated

Abstract

It is shown that within the class of all multivariate distributions depending on a location parameter (and satisfying certain smoothness conditions) and with a weighted norm constraint on the covariance matrix, the one with minimum Fisherian information is the Gaussian distribution. This result is then used in obtaining a tight upper bound on the error of estimating an unknown random vector observed in additive Gaussian noise under quadratic loss.
BibTeX
@article{basar1977optimum,
  author = {Tamer Başar},
  title = {{Optimum Fisherian Information for Multivariate Distributions}},
  journal = {The Annals of Statistics},
  year = {1977},
  doi = {10.1214/aos/1176344009},
}
Figure