On f-divergences Between Cauchy Distributions

Frank Nielsen, Kazuki OkamuraIEEE Trans. Information Theory 2021 · DOI · OpenReview

TL;DR

This paper reveals a surprising symmetry in $f$-divergences between Cauchy distributions, showing they can all be expressed as a function of the chi-squared divergence. This symmetry simplifies calculations and leads to new insights into their geometric properties, particularly for the Kullback-Leibler divergence.

What Problem Does This Paper Address?

Measuring the difference between probability distributions is fundamental in statistics and machine learning. $f$-divergences are a broad class of such measures, but their properties can be complex, especially for specific distributions. This paper focuses on the Cauchy distribution, a heavy-tailed distribution that appears in various fields but poses challenges due to its undefined mean and variance. The authors investigate whether $f$-divergences between Cauchy distributions exhibit special properties, particularly symmetry, and how these properties can be leveraged for analysis and geometric interpretation.

Methods

The core of the paper lies in proving the symmetry of $f$-divergences for Cauchy distributions. This is achieved through a combination of analytical techniques, including:

  • Invariance Properties: Leveraging the group structure of location-scale families and Möbius transformations to simplify calculations.
  • Maximal Invariants: Utilizing the concept of maximal invariants under group actions to establish relationships between different divergences.
  • Complex Analysis: Employing complex numbers and contour integration to derive closed-form expressions for divergences.
  • Taylor Series Expansions: Developing methods to express $f$-divergences as infinite series of power chi divergences, providing a way to approximate them.
  • Geometric Analysis: Investigating metrizations of divergences and their embedding properties in Hilbert spaces.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Universal Symmetry for Cauchy $f$-Divergences: A major finding is that all $f$-divergences between univariate Cauchy distributions are symmetric and can be expressed as a function of the chi-squared divergence. This is a significant simplification compared to other distributions.
  • Connection to Chi-Squared Divergence: The paper demonstrates that the symmetry arises from a specific relationship with the chi-squared divergence, which itself is symmetric for Cauchy distributions.
  • Metrizations and Hilbert Embeddings: The square roots of the Kullback-Leibler and Bhattacharyya divergences are shown to be distances, and the KL divergence is demonstrated to be isometrically embeddable into a Hilbert space. This has implications for geometric interpretations and potential applications in machine learning.
  • Multivariate Asymmetry: While univariate Cauchy $f$-divergences are symmetric, the paper shows that this symmetry does not generally hold for multivariate Cauchy distributions with different scale matrices.

Key Results

  • Symmetry Theorem: All $f$-divergences between univariate Cauchy distributions $p_\lambda$ and $p_{\lambda'}$ are symmetric and can be expressed as $I_f(p_\lambda: p_{\lambda'}) = h_f(\chi(\lambda, \lambda'))$, where $\chi$ is a specific function related to the parameters and $h_f$ is a strictly increasing function.
  • Closed-Form Expressions: Explicit formulas for the symmetry functions $h_f$ are provided for several important $f$-divergences, including Total Variation Distance, Kullback-Leibler Divergence, Jensen-Shannon Divergence, and LeCam-Vincze Divergence.
  • Taylor Series Expansions: Conditions are given for expanding $f$-divergences into infinite series of power chi divergences, with applications to Cauchy distributions.
  • Metric Properties: The square roots of the KL and Bhattacharyya divergences are shown to be distances on the upper-half plane.

Limitations and Future Work

  • Multivariate Asymmetry: The paper highlights that the symmetry observed for univariate Cauchy distributions does not extend to multivariate Cauchy distributions with different scale matrices, suggesting further research into the conditions for symmetry in multivariate settings.
  • Geodesic Properties: While the square roots of KL and Bhattacharyya divergences are shown to be distances, they are not Riemannian distances, and the spaces are not geodesic metric spaces. This opens avenues for exploring alternative geometric structures.
  • General $f$-Divergences: The paper focuses on specific $f$-divergences. Future work could explore the symmetry and metrization of a broader range of $f$-divergences for Cauchy distributions.
  • Computational Aspects: The paper includes a code snippet for Taylor expansions, suggesting that practical implementation and numerical exploration of these results are of interest.

Summary updated

Abstract

We prove that all f-divergences between univariate Cauchy distributions are symmetric, and can be expressed as functions of the chi-squared divergence. This property does not hold anymore for multivariate Cauchy distributions.
BibTeX
@article{nielsen2021f,
  author = {Frank Nielsen and Kazuki Okamura},
  title = {{On f-divergences Between Cauchy Distributions}},
  journal = {IEEE Trans. Information Theory},
  year = {2021},
  doi = {10.1109/tit.2022.3231645},
}
Figure