Abstract
The $k$-sample testing problem tests whether or not $k$ groups of data points are sampled from the same distribution. Multivariate analysis of variance (MANOVA) is currently the gold standard for $k$-sample testing but makes strong, often inappropriate, parametric assumptions. Moreover, independence testing and $k$-sample testing are tightly related, and there are many nonparametric multivariate independence tests with strong theoretical and empirical properties, including distance correlation (Dcorr) and Hilbert-Schmidt-Independence-Criterion (Hsic). We prove that universally consistent independence tests achieve universally consistent $k$-sample testing and that $k$-sample statistics like Energy and Maximum Mean Discrepancy (MMD) are exactly equivalent to Dcorr. Empirically evaluating these tests for $k$-sample scenarios demonstrates that these nonparametric independence tests typically outperform MANOVA, even for Gaussian distributed settings. Finally, we extend these non-parametric $k$-sample testing procedures to perform multiway and multilevel tests. Thus, we illustrate the existence of many theoretically motivated and empirically performant $k$-sample tests.
Introduction
Determining whether multiple groups of data originate from the same underlying distribution is a fundamental challenge in statistics. While Multivariate Analysis of Variance (MANOVA) has long been the standard approach, its reliance on Gaussian assumptions and uniform covariance structures often limits its utility in modern, complex datasets. This paper proposes a robust alternative by reframing the $k$-sample problem as an independence testing problem. By utilizing nonparametric tests such as Distance Correlation (Dcorr) and Multiscale Graph Correlation (MGC), the authors provide a framework that maintains high statistical power across diverse, high-dimensional, and nonlinear scenarios.
Methodology: $k$-Sample Testing as Independence Testing
The core insight of this work is that $k$-sample testing can be viewed as a special case of independence testing. If a dataset $X$ is independent of its group labels $Y$, the groups must be identically distributed. To implement this, the authors transform group labels into a structured "label matrix" (e.g., one-hot encoding for simple $k$-sample tests). This allows the application of advanced independence tests:
- Distance Correlation (Dcorr): Captures associations by analyzing distances between all pairs of points.
- Hilbert-Schmidt Independence Criterion (HSIC): A kernel-based method identifying dependencies in high-dimensional feature spaces.
- Multiscale Graph Correlation (MGC): An adaptive test searching across local and global scales to identify the strongest relationships.
- Kernel Mean Embedding Random Forest (KMERF): Utilizes random forest proximity structures to define similarity kernels.
This framework is further extended to accommodate multiway designs (simultaneous factor testing) and multilevel designs (nested or repeated measures) through block permutation strategies.
Theoretical Unification and Consistency
The authors establish a formal link between previously disparate statistical methods. They prove that $k$-sample Energy statistics are mathematically equivalent to Dcorr, and Maximum Mean Discrepancy (MMD) is equivalent to HSIC. Furthermore, they demonstrate that any "universally consistent" independence test—one capable of detecting any form of dependence as sample size increases—is also a universally consistent $k$-sample test. This theoretical grounding ensures that these nonparametric methods are not just heuristic alternatives but mathematically rigorous tools.
Empirical Results
The proposed methods were evaluated against MANOVA using synthetic Gaussian data, a benchmark suite of 20 diverse 2D distributions, high-dimensional simulations, and real-world fMRI data.
Gaussian Parity and Robustness
In ideal Gaussian settings where MANOVA's assumptions are met, nonparametric tests like Dcorr and MGC perform with comparable power, indicating no significant penalty for their use in standard scenarios.
Performance in Complex and High-Dimensional Data
As dimensionality increases, traditional MANOVA typically fails, especially when the number of dimensions approaches the sample size. In contrast, MGC and Dcorr maintain significant power. Across nonlinear benchmarks, MGC and KMERF consistently outperformed MANOVA, successfully detecting differences in complex distributional shapes.
Real-World Application: fMRI Analysis
The utility of the multilevel Dcorr approach was demonstrated on an fMRI dataset comparing meditators and novices. The method successfully identified significant differences in brain activity patterns across mental states while appropriately accounting for the nested structure of multiple scans per subject.
Conclusion
This work provides a unified, nonparametric framework for $k$-sample testing that overcomes the limitations of traditional MANOVA. By leveraging the flexibility of label matrices and the power of modern independence tests, researchers can achieve superior performance in high-dimensional and non-Gaussian settings. The authors have made these tools accessible via the hyppo Python package.
Summary updated