The Exact Equivalence of Independence Testing and Two-Sample Testing

Cencheng Shen, Carey E. Priebe, Joshua T. Vogelstein

Abstract

The $k$-sample testing problem tests whether or not $k$ groups of data points are sampled from the same distribution. Multivariate analysis of variance (MANOVA) is currently the gold standard for $k$-sample testing but makes strong, often inappropriate, parametric assumptions. Moreover, independence testing and $k$-sample testing are tightly related, and there are many nonparametric multivariate independence tests with strong theoretical and empirical properties, including distance correlation (Dcorr) and Hilbert-Schmidt-Independence-Criterion (Hsic). We prove that universally consistent independence tests achieve universally consistent $k$-sample testing and that $k$-sample statistics like Energy and Maximum Mean Discrepancy (MMD) are exactly equivalent to Dcorr. Empirically evaluating these tests for $k$-sample scenarios demonstrates that these nonparametric independence tests typically outperform MANOVA, even for Gaussian distributed settings. Finally, we extend these non-parametric $k$-sample testing procedures to perform multiway and multilevel tests. Thus, we illustrate the existence of many theoretically motivated and empirically performant $k$-sample tests. A Python package with all independence and k-sample tests called hyppo is available from https://hyppo.neurodata.io/.

Introduction

In statistical analysis, researchers frequently encounter two fundamental questions: $k$-sample testing (determining if multiple groups share the same underlying distribution) and independence testing (determining if two random variables are related). Traditionally, these have been treated as distinct problems with separate methodologies. Classical $k$-sample tests like ANOVA or MANOVA are widely used but often fail when data violates Gaussian assumptions or when differences exist in higher-order moments like variance rather than just the mean. This paper establishes a mathematical bridge between these fields, proving that independence measures can serve as powerful, universally consistent $k$-sample tests.

Methodology: The Population Transformation

The authors introduce a "Population Transformation" that converts the $k$-sample problem into an independence problem. By stacking data points from all $k$ groups into a single matrix $U$ and creating a corresponding matrix $V$ using one-hot encoding for group labels, the researchers prove that the groups have identical distributions if and only if $U$ is independent of $V$.

This unification demonstrates that several popular non-parametric tests are mathematically identical to dependence measures:

  • Distance Covariance (Dcov) is exactly equivalent to the Energy statistic.
  • Hilbert-Schmidt Independence Criterion (HSIC) is exactly equivalent to Maximum Mean Discrepancy (MMD).
  • For multiple groups, Dcov is equivalent to the Distance Components (DISCO) test.

Key Results

The proposed approach was evaluated using univariate Gaussian simulations and complex multivariate settings (e.g., spirals and circles). The results indicate that dependence-based tests are significantly more robust than classical parametric methods.

Figure 1: Comparison of testing power for ANOVA, Dcov, and HSIC across three scenarios: Size Difference (left), Mean Difference (center), and Variance Difference (right).

As shown in Figure 1, while ANOVA (red) is effective at detecting mean differences, it is entirely unable to detect differences in variance. Conversely, Dcov (blue) and HSIC (green) successfully detect both mean and variance differences, with testing power approaching 1.0 as the effect size increases. In multivariate tests, these measures also successfully identified distributional rotations that linear methods like MANOVA missed entirely.

Discussion and Future Work

This work provides a recipe for building robust, universally consistent tests capable of detecting any distributional difference given sufficient data. While the computational cost of permutation tests used to calculate p-values can be high for large datasets, the theoretical unification simplifies the choice of statistical tools. Future research will focus on computational optimizations and exploring which specific dependence measures are best suited for different data geometries. All methods discussed are implemented in the open-source hyppo Python package.

Summary updated

Abstract

The $k$-sample testing problem tests whether or not $k$ groups of data points are sampled from the same distribution. Multivariate analysis of variance (MANOVA) is currently the gold standard for $k$-sample testing but makes strong, often inappropriate, parametric assumptions. Moreover, independence testing and $k$-sample testing are tightly related, and there are many nonparametric multivariate independence tests with strong theoretical and empirical properties, including distance correlation (Dcorr) and Hilbert-Schmidt-Independence-Criterion (Hsic). We prove that universally consistent independence tests achieve universally consistent $k$-sample testing and that $k$-sample statistics like Energy and Maximum Mean Discrepancy (MMD) are exactly equivalent to Dcorr. Empirically evaluating these tests for $k$-sample scenarios demonstrates that these nonparametric independence tests typically outperform MANOVA, even for Gaussian distributed settings. Finally, we extend these non-parametric $k$-sample testing procedures to perform multiway and multilevel tests. Thus, we illustrate the existence of many theoretically motivated and empirically performant $k$-sample tests. A Python package with all independence and k-sample tests called hyppo is available from https://hyppo.neurodata.io/.
BibTeX
@inproceedings{shen2024exact,
  author = {Cencheng Shen and Carey E. Priebe and Joshua T. Vogelstein},
  title = {{The Exact Equivalence of Independence Testing and Two-Sample Testing}},
  booktitle = {Statistics},
  year = {2024},
  doi = {10.1016/j.spl.2024.110278},
}
Figure