The Representation Jensen-Rényi Divergence

Jhoan Keider Hoyos Osorio, Oscar Skean, Austin J. Brockmeier, Luis G. Sánchez GiraldoICASSP 2021

Introduction

In machine learning, quantifying the difference between data distributions is a fundamental challenge. Whether evaluating generative models or detecting distribution shifts in production, researchers often rely on measures like the Kullback-Leibler (KL) or Jensen-Shannon (JS) divergences. However, these traditional metrics typically require explicit probability density estimation, which is notoriously difficult and computationally expensive in high-dimensional spaces.

Kernel methods, such as Maximum Mean Discrepancy (MMD), offer an alternative by embedding distributions into a Reproducing Kernel Hilbert Space (RKHS). While effective, MMD primarily focuses on the "mean element" of a distribution. Consequently, it may fail to distinguish between distributions that share a mean but differ in higher-order moments, such as variance or kurtosis, unless the kernel is meticulously tuned.

The JRD Framework

The authors introduce the Representation Jensen-Rényi Divergence (JRD), a novel kernel-based metric that leverages the full eigenvalue spectrum of Gram matrices. JRD is built upon three core pillars:

  1. Representation Entropy: Utilizing a matrix-based entropy analogous to Rényi’s $\alpha$-order entropy, calculated via the eigenvalues of a normalized similarity (Gram) matrix.
  2. Mutual Information as Divergence: JRD frames divergence as the mutual information between a data point and its source distribution label. If the label provides no information about the point, the distributions are considered identical.
  3. Spectral Sensitivity: Unlike MMD, which reduces a distribution to a single mean vector, JRD utilizes the entire spectrum of the kernel operator. The parameter $\alpha$ allows for tuning sensitivity to different parts of the spectrum, making it more robust in detecting differences in higher-order moments.

To ensure scalability for large datasets, the authors employ Random Fourier Features (RFF), reducing the $O(N^3)$ complexity of eigenvalue decomposition to a more manageable computational load.

Empirical Evaluation and Results

The paper evaluates JRD across synthetic benchmarks, generative modeling tasks (MNIST), and a large-scale study of 44 real-world imbalanced datasets from the KEEL repository.

Statistical Power

In two-sample tests where distributions shared identical means but differed in variance, JRD demonstrated significantly higher statistical power than MMD. As shown in the figure below, JRD (particularly with $\alpha=5$) maintains a high acceptance rate for the alternative hypothesis ($H_1$) as dimensionality increases, whereas MMD's performance is less robust in specific variance-shift scenarios.

Statistical power of JRD vs MMD across varying dimensions. Plot (b) highlights JRD's superior sensitivity to variance differences when means are identical.

Generative Modeling and Imbalanced Data

When applied as a regularizer in a Variational Autoencoder (JRD-VAE), the metric achieved a test accuracy of 98.02% on MNIST, slightly outperforming the MMD-VAE baseline.

In the context of imbalanced classification, JRD was used as a subsampling tool to select the most representative subset of a majority class. In a benchmark against 44 datasets, JRD-based subsampling proved statistically superior to five out of six state-of-the-art methods, including popular ensemble techniques like RUSBoost and SMOTEBagging.

Performance comparison of JRD and its RFF variants against state-of-the-art methods for imbalanced data classification across 44 datasets.

Limitations and Future Work

While JRD offers significant advantages in sensitivity, it introduces new hyperparameters, specifically the $\alpha$ parameter and kernel bandwidth ($\sigma$). Although heuristics like the "median trick" are helpful, automated tuning remains an area for future research. Additionally, while RFF improves scalability, the spectral calculations remain more intensive than the simple mean-based computations of MMD. Future work may explore the integration of JRD with learned kernels in deep learning architectures.

Summary updated

Abstract

We introduce a divergence measure between data distributions based on operators in reproducing kernel Hilbert spaces defined by kernels. The empirical estimator of the divergence is computed using the eigenvalues of positive definite Gram matrices that are obtained by evaluating the kernel over pairs of data points. The new measure shares similar properties to Jensen-Shannon divergence. Convergence of the proposed estimators follows from concentration results based on the difference between the ordered spectrum of the Gram matrices and the integral operators associated with the population quantities. The proposed measure of divergence avoids the estimation of the probability distribution underlying the data. Numerical experiments involving comparing distributions and applications to sampling unbalanced data for classification show that the proposed divergence can achieve state of the art results.
BibTeX
@inproceedings{osorio2021representation,
  author = {Jhoan Keider Hoyos Osorio and Oscar Skean and Austin J. Brockmeier and Luis G. Sánchez Giraldo},
  title = {{The Representation Jensen-Rényi Divergence}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
}
Figure