Feature Collapsing for Gaussian Process Variable Ranking

Isaac Sebenius, Topi Paananen, Aki VehtariAISTATS 2022

Gaussian Processes (GPs) are widely utilized in probabilistic modeling due to their high-quality predictions and robust uncertainty estimates. However, interpreting these models remains a challenge. Determining which input features drive a model's decisions is crucial for high-stakes applications like medicine or finance. While Automatic Relevance Determination (ARD) is the standard approach for ranking feature importance in GPs, it suffers from significant "nonlinear bias," often misidentifying complex, "wiggly" features as more important than simpler linear ones.

In their AISTATS 2022 paper, Sebenius et al. introduce Feature Collapsing (FC), a novel variable ranking method designed to eliminate these biases with minimal computational overhead.

The Problem of Nonlinear Bias

Standard GP interpretability relies on ARD length-scale hyperparameters. A short length-scale implies high sensitivity, leading to a high relevance ranking. However, this heuristic is flawed: it conflates a feature's actual contribution to the output variance with its functional complexity.

Visualizing features with increasing degrees of nonlinearity across sinusoidal, cubic, and exponential forms.

As shown in Figure 1, features can exhibit varying degrees of "wiggliness." Traditional sensitivity methods, which often estimate the partial derivative of the latent function, naturally assign higher scores to nonlinear functions because they have steeper local slopes. This results in an unfair ranking where a highly nonlinear feature is deemed more "relevant" than a linear feature, even if both contribute equally to the model's predictive power.

Proposed Method: Feature Collapsing (FC)

Feature Collapsing addresses this by shifting the focus from local derivatives to global impact. The process involves three main steps:

  1. The Collapse: For a given input, the feature under test is "collapsed" to its mean (zero in standardized data), while all other features remain unchanged.
  2. Impact Measurement: The method calculates the distance between the original predictive distribution and the "collapsed" distribution.
  3. Metric: The primary metric used is the Bhattacharyya distance, which measures the similarity between two probability distributions.

By collapsing a feature to a constant reference point rather than applying a small perturbation, FC estimates the effect of the latent function relative to a global baseline. This effectively "levels the playing field" between linear and nonlinear features.

Key Results: Eliminating Bias and Improving Efficiency

The authors evaluated FC against ARD and other sensitivity methods (like KL divergence and variance-based methods) using both synthetic toy datasets and UCI benchmarks.

Zero Bias on Synthetic Data

On a custom toy dataset where eight variables contribute equally to output variance but have different functional forms, FC was the only method to consistently identify all features as equally relevant.

Relevance rankings for Sinusoidal, Cubic, and Exponential features. FC (green) consistently tracks the 'True' relevance (black dashed line) across all levels of nonlinearity, while ARD (blue) and KL (red) show significant bias.

As illustrated in Figure 4, ARD and KL methods show a massive bias toward more nonlinear features (higher input indices). In contrast, FC maintains a near-perfect ranking, matching the true relevance regardless of the feature's "wiggliness."

Real-World Performance and Speed

On UCI benchmark datasets, FC outperformed ARD and KL in 6 out of 8 cases, identifying the most influential features more effectively. Furthermore, FC is computationally efficient, maintaining a complexity of $O(pn^3)$. In practical tests, FC processed datasets in less than a second, whereas more mathematically rigorous alternatives like submodel projection took several hours.

Conclusion and Future Work

Feature Collapsing provides a faster, fairer way to rank variables in Gaussian Processes. By mitigating nonlinear bias without the computational cost of projection methods, it offers a practical tool for model interpretation. Future research may address the challenges of highly correlated features and explore the application of pointwise relevance scores for local explanations in clinical or individualized settings.

Summary updated

BibTeX
@inproceedings{sebenius2022feature,
  author = {Isaac Sebenius and Topi Paananen and Aki Vehtari},
  title = {{Feature Collapsing for Gaussian Process Variable Ranking}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
}
Figure