Abstract
We propose a kernel-based partial permutation test (PPT) for checking the equality of functional relationships between response variables and covariates across different groups. The core intuition is to keep the projections of the response vector $\boldsymbol{Y}$ on the leading principal components of a kernel matrix fixed while permuting the projections on the remaining principal components. This method allows for various kernel choices, corresponding to different function classes under the null hypothesis. Our test is exactly valid in finite samples for linear or polynomial regression models with Gaussian noise and remains asymptotically valid for a wider class of functions, including those modeled by regularized Gaussian processes. We further develop test statistics using likelihood ratios between nested Gaussian Process Regression (GPR) models and propose computationally efficient algorithms using EM and Newton’s methods. Both simulations and real-world applications demonstrate the power and flexibility of the proposed test.
Introduction
In empirical research—ranging from clinical trials to econometrics—a fundamental question is whether a relationship between a covariate (e.g., drug dosage) and a response (e.g., recovery rate) remains consistent across different demographic or experimental groups. This is the problem of testing the equality of functional relationships.
Traditional approaches typically face a trade-off. Parametric methods, like the classical F-test, are powerful but require strict assumptions about the functional form (e.g., linearity). Conversely, nonparametric methods offer flexibility but often struggle with multivariate data or require very large sample sizes to achieve validity. Furthermore, many existing permutation tests fail in "Regression Discontinuity" scenarios where covariate distributions do not overlap between groups. The Kernel-based Partial Permutation Test (PPT) bridges this gap by providing a distribution-free method that is powerful, flexible, and valid even in small samples.
The Partial Permutation Framework
The innovation of the PPT lies in how it generates the null distribution. Instead of shuffling the entire response vector or group labels—which can destroy the underlying structure of the data—the PPT shuffles only the "noise."
1. Kernel Representation and Eigen-decomposition
The relationship is modeled using a kernel matrix $K$, which defines the space of possible functions (linear, polynomial, or infinite-dimensional like Gaussian). We perform an eigen-decomposition of this matrix: $$K = \sum_{j=1}^n \lambda_j \mathbf{v}_j \mathbf{v}_j^T$$ In most smooth functional relationships, the "signal" (the actual relationship) is captured by the leading eigenvectors (those with large eigenvalues $\lambda_j$), while the "noise" is relegated to the tail eigenvectors.
2. Shuffling the Noise, Not the Signal
The response vector $\boldsymbol{Y}$ is projected onto these eigenvectors. The PPT keeps the projections on the leading eigenvectors fixed—preserving the smooth underlying function—and permutes only the projections onto the remaining eigenvectors. This creates a null distribution that maintains the functional signal while accounting for exchangeable noise components.
3. Computational Efficiency
To make the test practical, the authors utilize Likelihood Ratio Tests (LRT) based on Gaussian Process Regression (GPR). They implement an Expectation-Maximization (EM) algorithm and a Newton-based method (utilizing Fisher scoring and quadratic programming) to estimate variance components. This ensures the test can be performed hundreds of times for a permutation distribution without prohibitive computational costs.
Theoretical and Practical Strengths
- Exact Validity: For linear and polynomial kernels, the test is exactly valid in finite samples. Researchers do not need to rely on large-sample asymptotics for the p-values to be trustworthy.
- Handling Discontinuity: Because the test permutes projections rather than swapping group labels, it remains valid even when covariate distributions between groups are entirely different (e.g., comparing a treatment group with $X \in [0, 1]$ to a control group with $X \in [2, 3]$).
- Adaptability: By employing multiple kernels simultaneously, the test can inherit the strengths of different models—for instance, remaining exactly valid for linear components while staying flexible enough to capture non-linear residuals.
Empirical Results
Simulation Studies
The authors tested the PPT across various scenarios, including smooth vs. non-smooth functions (like the "sawtooth" function) and correlated vs. independent noise. In linear scenarios, the PPT performed nearly as well as the gold-standard F-test. However, when the true relationship was non-linear, the PPT significantly outperformed the F-test, which lost power by failing to account for the complexity of the data.
Real-World Application: Dutch Household Expenditure
The test was applied to a dataset comparing total monthly expenditure to food expenditure across households of different sizes. The PPT confirmed that "no-kid" households (size 2) have a significantly different food-spending curve than larger families. Interestingly, the test found no significant difference between families of size 3 and size 4, suggesting a plateau in how family size influences the food-to-total expenditure ratio (Engel curves).
Conclusion and Future Directions
The Kernel-based Partial Permutation Test offers a robust, computationally efficient alternative to traditional parametric and nonparametric tests. While it excels in low-to-moderate dimensions, the authors note that the "permutation size" (the amount of noise available to shuffle) can decrease as the number of covariates increases. Future work may involve incorporating sparsity assumptions to help the test scale to high-dimensional data and exploring non-Gaussian functional priors to further expand the test's applicability.
Summary updated