Kernel-Based Partial Permutation Test for Detecting Heterogeneous Functional Relationship

Xinran Li, Bo Jiang, Jun S. Liu

Abstract

We propose a kernel-based partial permutation test (PPT) for checking the equality of functional relationships between response variables and covariates across different groups. The core intuition is to keep the projections of the response vector $\boldsymbol{Y}$ on the leading principal components of a kernel matrix fixed while permuting the projections on the remaining principal components. This method allows for various kernel choices, corresponding to different function classes under the null hypothesis. Our test is exactly valid in finite samples for linear or polynomial regression models with Gaussian noise and remains asymptotically valid for a wider class of functions, including those modeled by regularized Gaussian processes. We further develop test statistics using likelihood ratios between nested Gaussian Process Regression (GPR) models and propose computationally efficient algorithms using EM and Newton’s methods. Both simulations and real-world applications demonstrate the power and flexibility of the proposed test.

Introduction

In empirical research—ranging from clinical trials to econometrics—a fundamental question is whether a relationship between a covariate (e.g., drug dosage) and a response (e.g., recovery rate) remains consistent across different demographic or experimental groups. This is the problem of testing the equality of functional relationships.

Traditional approaches typically face a trade-off. Parametric methods, like the classical F-test, are powerful but require strict assumptions about the functional form (e.g., linearity). Conversely, nonparametric methods offer flexibility but often struggle with multivariate data or require very large sample sizes to achieve validity. Furthermore, many existing permutation tests fail in "Regression Discontinuity" scenarios where covariate distributions do not overlap between groups. The Kernel-based Partial Permutation Test (PPT) bridges this gap by providing a distribution-free method that is powerful, flexible, and valid even in small samples.

The Partial Permutation Framework

The innovation of the PPT lies in how it generates the null distribution. Instead of shuffling the entire response vector or group labels—which can destroy the underlying structure of the data—the PPT shuffles only the "noise."

1. Kernel Representation and Eigen-decomposition

The relationship is modeled using a kernel matrix $K$, which defines the space of possible functions (linear, polynomial, or infinite-dimensional like Gaussian). We perform an eigen-decomposition of this matrix: $$K = \sum_{j=1}^n \lambda_j \mathbf{v}_j \mathbf{v}_j^T$$ In most smooth functional relationships, the "signal" (the actual relationship) is captured by the leading eigenvectors (those with large eigenvalues $\lambda_j$), while the "noise" is relegated to the tail eigenvectors.

2. Shuffling the Noise, Not the Signal

The response vector $\boldsymbol{Y}$ is projected onto these eigenvectors. The PPT keeps the projections on the leading eigenvectors fixed—preserving the smooth underlying function—and permutes only the projections onto the remaining eigenvectors. This creates a null distribution that maintains the functional signal while accounting for exchangeable noise components.

3. Computational Efficiency

To make the test practical, the authors utilize Likelihood Ratio Tests (LRT) based on Gaussian Process Regression (GPR). They implement an Expectation-Maximization (EM) algorithm and a Newton-based method (utilizing Fisher scoring and quadratic programming) to estimate variance components. This ensures the test can be performed hundreds of times for a permutation distribution without prohibitive computational costs.

Theoretical and Practical Strengths

  • Exact Validity: For linear and polynomial kernels, the test is exactly valid in finite samples. Researchers do not need to rely on large-sample asymptotics for the p-values to be trustworthy.
  • Handling Discontinuity: Because the test permutes projections rather than swapping group labels, it remains valid even when covariate distributions between groups are entirely different (e.g., comparing a treatment group with $X \in [0, 1]$ to a control group with $X \in [2, 3]$).
  • Adaptability: By employing multiple kernels simultaneously, the test can inherit the strengths of different models—for instance, remaining exactly valid for linear components while staying flexible enough to capture non-linear residuals.

Empirical Results

Simulation Studies

The authors tested the PPT across various scenarios, including smooth vs. non-smooth functions (like the "sawtooth" function) and correlated vs. independent noise. In linear scenarios, the PPT performed nearly as well as the gold-standard F-test. However, when the true relationship was non-linear, the PPT significantly outperformed the F-test, which lost power by failing to account for the complexity of the data.

Real-World Application: Dutch Household Expenditure

The test was applied to a dataset comparing total monthly expenditure to food expenditure across households of different sizes. The PPT confirmed that "no-kid" households (size 2) have a significantly different food-spending curve than larger families. Interestingly, the test found no significant difference between families of size 3 and size 4, suggesting a plateau in how family size influences the food-to-total expenditure ratio (Engel curves).

Conclusion and Future Directions

The Kernel-based Partial Permutation Test offers a robust, computationally efficient alternative to traditional parametric and nonparametric tests. While it excels in low-to-moderate dimensions, the authors note that the "permutation size" (the amount of noise available to shuffle) can decrease as the number of covariates increases. Future work may involve incorporating sparsity assumptions to help the test scale to high-dimensional data and exploring non-Gaussian functional priors to further expand the test's applicability.

Summary updated

Abstract

We propose a kernel-based partial permutation test for checking the equality of functional relationship between response and covariates among different groups. The main idea, which is intuitive and easy to implement, is to keep the projections of the response vector $\boldsymbol{Y}$ on leading principle components of a kernel matrix fixed and permute $\boldsymbol{Y}$'s projections on the remaining principle components. The proposed test allows for different choices of kernels, corresponding to different classes of functions under the null hypothesis. First, using linear or polynomial kernels, our partial permutation tests are exactly valid in finite samples for linear or polynomial regression models with Gaussian noise; similar results straightforwardly extend to kernels with finite feature spaces. Second, by allowing the kernel feature space to diverge with the sample size, the test can be large-sample valid for a wider class of functions. Third, for general kernels with possibly infinite-dimensional feature space, the partial permutation test is exactly valid when the covariates are exactly balanced across all groups, or asymptotically valid when the underlying function follows certain regularized Gaussian processes. We further suggest test statistics using likelihood ratio between two (nested) GPR models, and propose computationally efficient algorithms utilizing the EM algorithm and Newton's method, where the latter also involves Fisher scoring and quadratic programming and is particularly useful when EM suffers from slow convergence. Extensions to correlated and non-Gaussian noises have also been investigated theoretically or numerically. Furthermore, the test can be extended to use multiple kernels together and can thus enjoy properties from each kernel. Both simulation study and application illustrate the properties of the proposed test.
BibTeX
@article{li2021kernel,
  author = {Xinran Li and Bo Jiang and Jun S. Liu},
  title = {{Kernel-Based Partial Permutation Test for Detecting Heterogeneous Functional Relationship}},
  journal = {Journal of the American Statistical Association},
  year = {2021},
  doi = {10.1080/01621459.2021.2000867},
}
Figure