Introduction
In machine learning, quantifying the difference between data distributions is a fundamental challenge. Whether evaluating generative models or detecting distribution shifts in production, researchers often rely on measures like the Kullback-Leibler (KL) or Jensen-Shannon (JS) divergences. However, these traditional metrics typically require explicit probability density estimation, which is notoriously difficult and computationally expensive in high-dimensional spaces.
Kernel methods, such as Maximum Mean Discrepancy (MMD), offer an alternative by embedding distributions into a Reproducing Kernel Hilbert Space (RKHS). While effective, MMD primarily focuses on the "mean element" of a distribution. Consequently, it may fail to distinguish between distributions that share a mean but differ in higher-order moments, such as variance or kurtosis, unless the kernel is meticulously tuned.
The JRD Framework
The authors introduce the Representation Jensen-Rényi Divergence (JRD), a novel kernel-based metric that leverages the full eigenvalue spectrum of Gram matrices. JRD is built upon three core pillars:
- Representation Entropy: Utilizing a matrix-based entropy analogous to Rényi’s $\alpha$-order entropy, calculated via the eigenvalues of a normalized similarity (Gram) matrix.
- Mutual Information as Divergence: JRD frames divergence as the mutual information between a data point and its source distribution label. If the label provides no information about the point, the distributions are considered identical.
- Spectral Sensitivity: Unlike MMD, which reduces a distribution to a single mean vector, JRD utilizes the entire spectrum of the kernel operator. The parameter $\alpha$ allows for tuning sensitivity to different parts of the spectrum, making it more robust in detecting differences in higher-order moments.
To ensure scalability for large datasets, the authors employ Random Fourier Features (RFF), reducing the $O(N^3)$ complexity of eigenvalue decomposition to a more manageable computational load.
Empirical Evaluation and Results
The paper evaluates JRD across synthetic benchmarks, generative modeling tasks (MNIST), and a large-scale study of 44 real-world imbalanced datasets from the KEEL repository.
Statistical Power
In two-sample tests where distributions shared identical means but differed in variance, JRD demonstrated significantly higher statistical power than MMD. As shown in the figure below, JRD (particularly with $\alpha=5$) maintains a high acceptance rate for the alternative hypothesis ($H_1$) as dimensionality increases, whereas MMD's performance is less robust in specific variance-shift scenarios.
Generative Modeling and Imbalanced Data
When applied as a regularizer in a Variational Autoencoder (JRD-VAE), the metric achieved a test accuracy of 98.02% on MNIST, slightly outperforming the MMD-VAE baseline.
In the context of imbalanced classification, JRD was used as a subsampling tool to select the most representative subset of a majority class. In a benchmark against 44 datasets, JRD-based subsampling proved statistically superior to five out of six state-of-the-art methods, including popular ensemble techniques like RUSBoost and SMOTEBagging.
Limitations and Future Work
While JRD offers significant advantages in sensitivity, it introduces new hyperparameters, specifically the $\alpha$ parameter and kernel bandwidth ($\sigma$). Although heuristics like the "median trick" are helpful, automated tuning remains an area for future research. Additionally, while RFF improves scalability, the spectral calculations remain more intensive than the simple mean-based computations of MMD. Future work may explore the integration of JRD with learned kernels in deep learning architectures.
Summary updated