Imagine you are handed two large datasets of high-dimensional images—one collected from a hospital's primary scanner and another from a newly deployed backup scanner. You want to know if the two scanners produce identical data distributions, or if a subtle distribution shift has occurred.
In classical statistics, you would turn to a two-sample test like the Maximum Mean Discrepancy (MMD) or a Wasserstein distance test. You feed in both datasets, and the test spits out a single number: a $p$-value or a global distance metric. If the $p$-value is close to zero, the null hypothesis is rejected, and you know the distributions differ.
But this leaves you with a frustrating black box. As Cynthia Rudin et al. point out, standard quantitative metrics are opaque; they tell you that a change occurred, but fail to explain which features or data subgroups are responsible. In practice, knowing a shift exists is only half the battle—you need to know exactly where and why the distributions diverge to diagnose the underlying issue.
To bridge this gap, a major research direction has emerged focusing on interpretable statistical two-sample testing. Instead of treating distribution comparison as a purely global, scalar-valued problem, these modern techniques optimize tests to explicitly localize, attribute, and explain feature-level discrepancies.
Learning to Discriminate as an Explanation
One of the most intuitive ways to find out how two datasets differ is to turn the statistical problem into a machine learning problem. This is the core philosophy behind Classifier Two-Sample Tests (C2ST), popularized by David Lopez-Paz et al..
The intuition is elegant: if you pool your two datasets, label dataset $A$ as class 0 and dataset $B$ as class 1, and train a binary classifier, what happens? If the distributions are identical, the classifier will do no better than a random coin flip (an accuracy of 0.5). If the classifier achieves high accuracy, the distributions are fundamentally different.
Dataset A (P) ───> [ Label 0 ] ┐
├─> Train Binary Classifier ─> High Accuracy? ─> Distributions Differ!
Dataset B (Q) ───> [ Label 1 ] ┘
│
└─> High-Confidence Predictions = Core Discrepancies
Crucially, this framework provides built-in interpretability. By examining the data points where the classifier has the highest classification confidence, you can pinpoint the exact regions of the feature space where the two distributions diverge most significantly.
While powerful, standard C2ST can be brittle if the underlying classifier overfits or is poorly calibrated. To resolve this, Vansh Bansal et al. introduced Conformal C2ST, which applies conformal calibration to the classifier's outputs. This framework guarantees exact finite-sample validity regardless of the classifier's complexity, turning even weak or biased models into rigorous, locally interpretable two-sample tests.
Optimizing Witness Points and Projections
Another way to make a test interpretable is to force the mathematical framework to pick out a small, human-examinable set of features or locations that maximize the visible difference between the two populations.
Witness Points
Instead of evaluating a distribution discrepancy across the entire high-dimensional space, Wittawat Jitkrittum et al. proposed a method that automatically identifies specific spatial or frequency locations—called witness points—where the two distributions differ the most.
By optimizing these witness points to maximize the statistical power of the test, the algorithm naturally surfaces the most informative regions. For example, if comparing text distributions, the optimized witness points might correspond to specific keywords; in computer vision, they highlight the exact visual regions where an image dataset deviates.
Informative Projections
High-dimensional spaces suffer from the "curse of dimensionality," where data becomes sparse and global distances lose their meaning. To combat this while preserving interpretability, Jie Wang et al. developed a test based on the Projected Wasserstein (PW) distance.
The core idea is to search for an optimal low-dimensional projection subspace that maximizes the Wasserstein distance between the two datasets. Because the test isolates the specific linear combinations of features that maximize the discrepancy, the projection vector itself serves as a direct, physical explanation of the distribution shift.
High-Dimensional Data (X, Y) ───[ Optimize Projection Vector v ]───> Maximize Distance(vᵀX, vᵀY)
│
└─> Vector coefficients explain WHICH features drive the shift
Similarly, Zhijian Zhou et al. leverage multiple Mahalanobis kernels to construct a bi-directional hypothesis test. This approach allows them to strictly control false positives while accurately pinpointing the precise local regions where two distributions diverge.
Feature Attribution via Game Theory and Kernels
Kernel-based methods, such as the Maximum Mean Discrepancy (MMD), map data into an infinite-dimensional Reproducing Kernel Hilbert Space (RKHS) to detect complex, non-linear differences. While deep kernels (Feng Liu et al.) and meta-learning extensions (Feng Liu et al.) have drastically increased the statistical power of these tests in low-data regimes, they traditionally offer no way to look under the hood.
To extract feature-level explanations from these complex kernel tests, researchers have begun merging statistical testing with cooperative game theory.
The breakthrough comes from attributing the global test statistic using Shapley values. If we treat the final MMD score as a "payout" achieved by a coalition of features, we can use Shapley values to fairly distribute the credit (or blame) for that score to each individual feature.
Historically, calculating exact Shapley values required exponential time relative to the number of features, forcing practitioners to rely on noisy approximations. However, Majid Mohammadi et al. developed PKeX-Shapley, an algorithm that computes exact Shapley values for product-kernel methods (like RBF or Laplacian kernels) in polynomial time. This math transforms global hypothesis testing into a transparent process, directly identifying the exact features driving a dataset discrepancy without approximation noise.
[ Dataset P ] vs [ Dataset Q ] ───> Global MMD Test ───> [ PKeX-Shapley ] ───> Exact Feature Attribution Scores
In parallel, researchers have modified the kernels themselves to perform feature selection during the testing process. For instance, Michael F. Adamer et al. introduced SpInOpt-MMD, a method that optimizes a kernel test to simultaneously assess global distribution differences and select sparse, distinguishing features. In biomedical applications, this allows the test to double as a biomarker identification tool, explaining the biological mechanism behind a detected disease state.
Why It Matters: Beyond Global Metrics
Shifting from global metrics to localized, interpretable testing fundamentally changes how we diagnose errors in machine learning and complex systems:
- Model Discrepancy Auditing: Even when two models achieve identical aggregate accuracy, they often suffer from predictive multiplicity—disagreeing on a significant number of individual predictions. Marcin Detyniecki et al. demonstrate that by generating localized "discrepancy intervals" in the feature space, practitioners can move beyond aggregate metrics to understand exactly where and why two models diverge in their behavior.
- Geometry-Aware Testing: By moving away from rigid Euclidean assumptions and utilizing alternative geometries—such as the $L^1$ geometry proposed by Meyer Scetbon et al. or domain-specific Laplacian eigenfunctions proposed by Raif M. Rustamov et al.—interpretable tests can distinguish whether group variations are caused by localized bursts of activity or broad, global shifts across complex domains like graphs and manifolds.
Ultimately, interpretable statistical two-sample testing turns hypothesis testing from a passive alarm system into an actionable diagnostic tool. By telling us exactly where distributions part ways, these methods give engineers and scientists the precise insights needed to debug models, discover features, and understand shifting data environments.
Key Foundational Work
Revisiting Classifier Two-Sample Tests (2017) — contributed 3 concept elements to this direction.
Interpretable Distribution Features with Maximum Testing Power (2016) — contributed 2 concept elements to this direction.