Permutation-Based Hypothesis Testing for Neural Networks

Francesca Mandel, Ian BarnettAAAI 2024 · DOI

Introduction

Neural networks have revolutionized predictive modeling but are frequently criticized as "black boxes." While they offer high accuracy, they often lack the statistical rigor and interpretability of traditional models like linear regression. This paper introduces a novel permutation-based framework designed to perform formal hypothesis testing on neural network features. By calculating p-values, researchers can now determine whether a feature’s association with the output is statistically significant and distinguish between linear and nonlinear relationships.

The Need for Statistical Inference in Deep Learning

In high-stakes domains such as medicine, criminal justice, and genomics, understanding the "why" behind a prediction is as crucial as the prediction itself. Existing interpretability methods like SHAP or LIME provide feature importance rankings but do not offer a path to classical statistical inference. There is no standard "p-value" for a neural network's internal representations.

The authors address this gap by moving beyond heuristic rankings to formal hypothesis testing. They focus on two fundamental questions:

  1. Association: Is a specific input feature truly related to the output, or is the model capturing random noise?
  2. Nonlinearity: If an association exists, is it a simple linear relationship, or has the network learned a complex, nonlinear pattern?

Methodology: Partial Derivatives and Permutation

The core of the proposed approach lies in the use of partial derivatives. By calculating the derivative of the network output with respect to a specific input, the authors quantify the model's "sensitivity" to that feature.

Testing for Association

To test if a feature $X_j$ is significantly associated with the output, the framework employs a permutation test. The values of $X_j$ are shuffled while keeping other features constant, and the network is retrained multiple times. If the original network's sensitivity to $X_j$ is significantly higher than the sensitivities observed in the shuffled datasets, the association is deemed statistically significant.

Testing for Nonlinearity

A purely linear relationship is characterized by a constant partial derivative. To detect nonlinearity, the authors analyze the residuals of the partial derivatives. They fit a Generalized Additive Model (GAM) to these residuals; if the GAM identifies a significant pattern, it indicates that the neural network has captured a nonlinear relationship. This architecture-agnostic approach can be applied to any feed-forward structure with differentiable activation functions (e.g., Sigmoid or ReLU).

Experimental Validation

The framework was validated across three distinct data environments:

  1. Simulated Data: Five types of associations (Linear, Quadratic, Cubic, Trigonometric, and Nonsmooth) were used to evaluate the test's statistical power and Type-I error rate.
  2. Pediatric Concussion Data: A dataset of 544 observations from the Children’s Hospital of Philadelphia (CHOP) was analyzed to link clinical symptoms and device-based metrics to concussion scores.
  3. Parkinson’s Disease Genomic Data: Genomic data from the AMP PD project was used to test associations between specific genes and disease status.
Figure 1: Comparison of statistical power across different association types. Neural networks (NN-1, NN-2) show a distinct advantage in detecting "nonsmooth" relationships compared to traditional GAM and LM approaches.

Key Results and Insights

The study yielded several significant findings:

  • High Statistical Power: In simulations, the nonlinearity test achieved a power of 1.00 for quadratic and cubic relationships, meaning it consistently identified nonlinear signals when present.
  • The "Nonsmooth" Advantage: A standout result is the neural network's ability to handle "nonsmooth" associations—relationships with sharp jumps or jagged edges. While traditional models like Linear Models and GAMs struggle with these patterns, the neural network testing framework excels at detecting them.
  • Real-World Discovery: In the pediatric concussion study, the framework confirmed that "symptom severity" has a highly significant nonlinear association with the total symptom score ($p < 0.001$). In contrast, symptoms like "headache" and "dizziness" exhibited strong linear associations.
Figure 5: P-values for nonlinearity and association tests applied to the Pediatric Concussion Data.

Discussion and Future Work

This research demonstrates that we do not have to sacrifice interpretability for predictive power. By providing a path to p-values, neural networks can offer the same inferential clarity as traditional statistical models.

However, challenges remain. The primary limitation is computational cost. Permutation testing requires retraining the network hundreds or thousands of times, which is resource-intensive for large-scale deep learning models. Furthermore, the current association test assumes independence between predictors; high correlation between features can lead to increased Type-I error rates. Future research will focus on developing "conditional" permutation methods to better handle correlated data structures and improving computational efficiency.

Summary updated

BibTeX
@inproceedings{mandel2024permutation,
  author = {Francesca Mandel and Ian Barnett},
  title = {{Permutation-Based Hypothesis Testing for Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i13.29343},
}
Figure