A framework for paired-sample hypothesis testing for high-dimensional data

Ioannis Bargiotas, Argyris Kalogeratos, Nicolas VayatisarXiv 2023 · arXiv

TL;DR

This paper introduces the Multivariate Wilcoxon Signed-Rank (MWSR) test, a new framework for paired-sample testing in high-dimensional settings. By transforming the problem into a classification task using an aggregated "pseudomedian hyperplane," it achieves significantly higher sensitivity than traditional methods while naturally identifying which features contribute most to the observed changes.

What Problem Does This Paper Address?

Researchers across fields like medicine and finance frequently need to compare paired data—such as a patient cohort measured before and after a specific treatment—across dozens or even hundreds of features.

Traditional statistical approaches face a dilemma in these high-dimensional settings. One common method, Multiple Testing (MT), performs a separate test for every feature. However, to avoid false positives (Type I errors), researchers must apply aggressive p-value adjustments like the Bonferroni correction, which severely reduces the statistical power to detect real changes. Alternatively, multivariate tests like Hotelling’s $T^2$ often struggle when the number of features approaches or exceeds the number of samples ($d \ge N$), or when the data doesn't follow a perfect normal distribution. The authors aim to bridge this gap by creating a non-parametric multivariate test that remains powerful and interpretable in high dimensions.

Methods

The authors propose a two-step procedure called the Multivariate Wilcoxon Signed-Rank (MWSR) test. The core idea is to treat hypothesis testing as a classification problem.

  1. Constructing a Scoring Function: In a single dimension, the midpoint between two paired points acts as a simple decision rule. The authors generalize this to $d$-dimensions: for every pair of instances, they calculate a perpendicular bisecting hyperplane. This hyperplane perfectly separates the "before" and "after" points for that specific pair.
  2. Aggregation via Pseudomedian: To create a single, robust rule for the entire dataset, they aggregate these individual hyperplanes. They extend the classical Hodges-Lehmann estimator to find a "pseudomedian hyperplane" by calculating the Walsh averages (pairwise midpoints) of all the individual hyperplane coefficients and intercepts.
  3. Testing: This final pseudomedian hyperplane is used to score every data point, effectively reducing the complex multidimensional data into a single dimension. Finally, a standard univariate Wilcoxon signed-rank (WSR) test is performed on these scores to determine if the shift between the two samples is statistically significant.
Figure 1: An illustrative example of how we extend the WSR paired-sample hypothesis testing from one dimension (a) to multiple dimensions with the MWSR test (b). We see each pair’s midpoint as a decision rule separating the data space, and we generalize this in multiple dimensions by means of perpendicular bisecting hyperplanes.

Datasets

The framework was evaluated using both simulated and real-world data:

  • Synthetic Datasets: The authors simulated Gaussian distributions with $N=30$ pairs and varying dimensions ($d = 10, 20, 30, 60$). In these sets, 90% of the features were pure noise, while only 10% contained a linear shift.
  • Real Clinical Dataset: This involved posturographic (balance) assessment data for 30 subjects with Parkinson’s syndrome. The data consisted of 16 features (like velocity and acceleration of body sway) measured at two time points, six months apart.

Key Insights and Ideas

  • Classification as a Proxy: The paper reinforces the idea that if a classifier can successfully distinguish between two groups, there is likely a statistically significant difference between them.
  • Robustness to the "Curse of Dimensionality": Unlike Hotelling’s $T^2$, which requires inverting a covariance matrix and fails entirely when $d \ge N$, the MWSR approach remains stable and effective even when features outnumber samples.
  • Built-in Interpretability: Because the scoring function is a linear hyperplane, the coefficients (weights) assigned to each feature provide an immediate "feature importance" index. This tells the researcher exactly which variables are driving the detected change.

Key Results

  • Superior Sensitivity: In synthetic trials, MWSR was consistently more sensitive than both Multiple Testing (with Bonferroni) and Hotelling’s $T^2$, detecting much smaller shifts in mean values.
  • Performance in High Dimensions: As shown in the figure below, the performance gap between MWSR and traditional methods widens as dimensionality increases. Notably, when $d \ge 30$ (where $d \ge N$), Hotelling’s $T^2$ fails to detect any significant results, while MWSR maintains high power.
Figure 2: The average performance of paired-sample testing approaches in synthetic datasets with N = 30 pairs of instances coming from two Gaussian distributions, both with either std = 1 or 2 (first and second row, respectively). The performance is presented as a function of the separation distance between the two distributions (average difference on x-axis). On the y-axis it appears the % of significant results detected by each method, and on the x-axis the progressive difference in the mean value for the 10% of the dimensions (d).
  • Real-World Validation: On the Parkinson’s dataset, the MWSR test successfully detected a significant change in patients over the six-month period. In contrast, the traditional Multiple Testing approach failed to find any significant difference, likely due to the loss of power from p-value adjustments.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Linear Shifts Only: The current framework is designed specifically to detect linear shifts in location. It may not be suitable for detecting more complex, non-linear changes in the data distribution.
  • Sensitivity to Normalization: For the feature importance weights to be truly interpretable, the input data must be properly normalized.
  • Feature Correlation: The weight assigned to a specific feature can be influenced by its correlation with other features, which requires careful interpretation by the user.

Future work involves investigating the theoretical regimes where this method is qualitatively superior and exploring its efficiency in other experimental setups.

Summary updated

Abstract

The standard paired-sample testing approach in the multidimensional setting applies multiple univariate tests on the individual features, followed by p-value adjustments. Such an approach suffers when the data carry numerous features. A number of studies have shown that classification accuracy can be seen as a proxy for two-sample testing. However, neither theoretical foundations nor practical recipes have been proposed so far on how this strategy could be extended to multidimensional paired-sample testing. In this work, we put forward the idea that scoring functions can be produced by the decision rules defined by the perpendicular bisecting hyperplanes of the line segments connecting each pair of instances. Then, the optimal scoring function can be obtained by the pseudomedian of those rules, which we estimate by extending naturally the Hodges-Lehmann estimator. We accordingly propose a framework of a two-step testing procedure. First, we estimate the bisecting hyperplanes for each pair of instances and an aggregated rule derived through the Hodges-Lehmann estimator. The paired samples are scored by this aggregated rule to produce a unidimensional representation. Second, we perform a Wilcoxon signed-rank test on the obtained representation. Our experiments indicate that our approach has substantial performance gains in testing accuracy compared to the traditional multivariate and multiple testing, while at the same time estimates each feature's contribution to the final result.
BibTeX
@misc{bargiotas2023framework,
  author = {Ioannis Bargiotas and Argyris Kalogeratos and Nicolas Vayatis},
  title = {{A framework for paired-sample hypothesis testing for high-dimensional data}},
  year = {2023},
  eprint = {2309.16274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.16274},
}
Figure