Abstract
We construct and analyze a neural network two-sample test to determine whether two datasets came from the same distribution (null hypothesis) or not (alternative hypothesis). We perform time-analysis on a neural tangent kernel (NTK) two-sample test. In particular, we derive the theoretical minimum training time needed to ensure the NTK two-sample test detects a deviation-level between the datasets. Similarly, we derive the theoretical maximum training time before the NTK two-sample test detects a deviation-level. By approximating the neural network dynamics with the NTK dynamics, we extend this time-analysis to the realistic neural network two-sample test generated from time-varying training dynamics and finite training samples. A similar extension is done for the neural network two-sample test generated from time-varying training dynamics but trained on the population. To give statistical guarantees, we show that the statistical power associated with the neural network two-sample test goes to 1 as the neural network training samples and test evaluation samples go to infinity. Additionally, we prove that the training times needed to detect the same deviation-level in the null and alternative hypothesis scenarios are well-separated. Finally, we run some experiments showcasing a two-layer neural network two-sample test on a hard two-sample test problem and plot a heatmap of the statistical power of the two-sample test in relation to training time and network complexity.
Introduction
In statistical inference, the two-sample test is a foundational tool used to determine if two sets of observations originate from the same underlying distribution. While classical methods like the t-test or Maximum Mean Discrepancy (MMD) are effective for low-dimensional data, they often falter when faced with the high-dimensional, non-linear complexities of modern datasets in fields like genomics or finance. Neural networks, with their inherent ability to learn complex features, offer a promising alternative. However, their use in hypothesis testing is complicated by the risk of overfitting: a sufficiently large network might "learn" to distinguish two identical distributions simply by memorizing finite sample noise. This research provides the first rigorous theoretical guarantees for neural network-based two-sample tests by analyzing their training dynamics through the lens of Neural Tangent Kernels (NTK).
Methodology: Bridging Theory and Practice
The study bridges the gap between idealized theoretical models and practical neural network training by categorizing the analysis into three distinct regimes: Realistic, Population, and Auxiliary NTK.
- Auxiliary NTK: A theoretical baseline where the kernel remains fixed at its initial state.
- Population Dynamics: An "ideal" scenario involving training on an infinite data stream.
- Realistic Dynamics: The real-world scenario involving finite samples and time-varying gradients.
By employing a small-time approximation, the authors analyze the early stages of gradient descent. They demonstrate that the test statistic—defined as the difference in average network output between the two samples—evolves according to how the "witness function" (the difference between the two densities) projects onto the eigenfunctions of the NTK.
Theoretical Insights: The Spectral Shortcut and Detection Times
A key finding is the "Spectral Shortcut": the network does not need to learn the entire distribution to perform a test. Instead, the detection time depends on the subset of the NTK spectrum that the witness function projects onto. Large-scale, "low-frequency" differences between distributions are detected almost immediately.
Crucially, the research proves that the time required to detect a genuine signal (under the alternative hypothesis $H_1$) is significantly shorter than the time it takes for the network to overfit and "detect" a difference in identical distributions (under the null hypothesis $H_0$).
This separation creates a "sweet spot" in training time where the test is both sensitive and reliable. The authors derive explicit lower bounds on the time difference between $t^+(\epsilon)$ (time to detect noise) and $t^-(\epsilon)$ (time to detect signal), showing that with high probability, $t^+(\epsilon) > t^-(\epsilon)$.
Results and Empirical Validation
The theoretical framework is validated using synthetic datasets, including Multivariate Normals and complex Gaussian Mixture Models (GMM). The GMMs represent a "hard" test case where distributions have overlapping regions, making them difficult for linear tests.
Key results include:
- Asymptotic Statistical Power: The authors prove that as the number of training and test samples approaches infinity, the statistical power of the neural network two-sample test converges to 1.
- Empirical Sensitivity: Experiments show that even a single gradient descent step significantly enhances the test's sensitivity compared to random initialization.
- Complexity Trade-offs: Heatmaps of statistical power reveal a relationship between training epochs and network complexity (parameter-to-sample ratio), mirroring "double descent" behavior.
Conclusion and Future Work
This work establishes a theoretical foundation for using neural networks in hypothesis testing. However, several avenues for future research remain. The current analysis is primarily situated in the "small-time" or "lazy training" regime where the NTK remains relatively static. Extending these guarantees to "active" training regimes, where the kernel evolves significantly, is a major open challenge. Additionally, while the theory is general, further exploration into deeper architectures and specialized layers (e.g., Convolutional or Attention) could refine the detection time constants for specific data modalities. Finally, optimizing the computational pipeline—currently involving full training and permutation tests—is necessary for practical, large-scale applications.
Summary updated