Abstract
Homogeneity testing is a fundamental statistical task aimed at determining if two samples originate from the same distribution. While Maximum Mean Discrepancy (MMD) has become a popular kernel-based approach for high-dimensional data, it suffers from degeneracy under the null hypothesis and reduced power in multiple-comparison scenarios. This paper introduces Regularized Maximum Mean Discrepancy (RMMD), which incorporates a regularization term to stabilize the test statistic. We demonstrate that RMMD converges to a normal distribution, simplifying threshold calculation, and that a regularization parameter of $\kappa=1$ is theoretically optimal. Empirical evaluations across synthetic and real-world datasets show that RMMD consistently outperforms standard MMD and other baselines in terms of statistical power and efficiency, particularly in small-sample regimes.
Introduction
In statistics and machine learning, the homogeneity test addresses whether two data samples are drawn from the same underlying distribution. While classical tests like Kolmogorov-Smirnov (KS) are effective for simple data, they often struggle with high-frequency features in complex, high-dimensional datasets. Modern approaches leverage Maximum Mean Discrepancy (MMD), which maps data into a Reproducing Kernel Hilbert Space (RKHS) for comparison.
However, MMD possesses a significant drawback: its statistical behavior "degenerates" under the null hypothesis (when distributions are identical). This degeneracy results in a complex null distribution, often requiring computationally expensive bootstrapping to determine significance thresholds. Furthermore, in multiple-comparison tasks—such as analyzing multi-channel EEG signals—the power of MMD can drop sharply, leading to missed detections of true differences.
Methods: Regularized Maximum Mean Discrepancy
To address these limitations, the authors propose Regularized Maximum Mean Discrepancy (RMMD). This method modifies the standard MMD squared distance by introducing a penalty based on the norm of the mean embeddings of the distributions.
Asymptotic Normality and Parameter Optimality
A key technical advantage of RMMD is its convergence to a simple Normal distribution under the null hypothesis, contrasting with the "infinite sum of chi-squares" distribution of standard MMD. This asymptotic normality allows for faster and more reliable calculation of significance thresholds without intensive resampling.
Furthermore, the authors provide a mathematical proof that setting the regularization constant $\kappa = 1$ is optimal for maximizing the power of the test. This theoretical finding simplifies practical application by removing the need for hyperparameter tuning.
Experimental Evaluation
The authors evaluated RMMD against standard MMD, Kernel Fisher Discriminant Analysis (KFDA), and traditional tests like KS across a variety of datasets, including synthetic periodic distributions, MNIST handwritten digits, and real-world EEG signals.
Key Results
In nearly every experimental setting, RMMD demonstrated superior statistical power. As shown in the performance summary below, RMMD consistently achieved the highest power scores, particularly on complex datasets like MNIST and Periodic distributions where traditional MMD and KS tests showed lower sensitivity.
Beyond raw power, RMMD showed significant advantages in:
- Sample Efficiency: Using the Asymptotic Relative Efficiency (ARE) metric, the authors confirmed that RMMD requires fewer samples than MMD to achieve the same level of statistical certainty.
- Multiple Comparison Control: In the EEG task, RMMD effectively isolated visual cortex activity with a lower False Discovery Rate (FDR) than MMD, which struggled with the noise inherent in multi-electrode data.
Conclusion
RMMD provides a robust and powerful alternative to standard MMD for homogeneity testing. By stabilizing the variance of the test statistic through regularization, it avoids the degeneracy issues of MMD and offers a predictable Normal null distribution. Its performance is particularly noteworthy in small-sample scenarios and complex multiple-comparison problems, making it a valuable tool for high-dimensional data analysis in fields like neuroscience and computer vision.
Summary updated