A simple approach for local and global variable importance in nonlinear regression models

Emily T. Winn-Nuñez, Maryclare Griffin, Lorin Crawford

TL;DR

The paper introduces GOALS (GlObal And Local Score), a new framework for interpreting nonlinear machine learning models by providing both individual-level (local) and population-level (global) variable importance scores simultaneously. It is computationally efficient, theoretically grounded in function space perturbations, and excels at identifying complex feature interactions that traditional linear-based interpretability methods often miss.

What Problem Does This Paper Address?

In the world of machine learning, there is a persistent trade-off between model performance and interpretability. Linear models are easy to understand (via coefficients), but they struggle with the nonlinear interactions common in complex biological data. Conversely, "black box" models like Gaussian Processes (GPs) and Neural Networks handle these complexities well but are notoriously difficult to explain.

Existing interpretability tools usually fall into two camps:

  1. Global Methods: These rank features based on their overall contribution to the population (e.g., RATE).
  2. Local Methods: These explain why a specific prediction was made for a specific individual (e.g., SHAP).

The authors identify a gap: we lack a unified, scalable approach that provides both perspectives while remaining robust to nonlinearities. This is especially critical in fields like precision medicine, where a researcher needs to know both which genes affect a disease generally and which specific markers drive a particular patient's prognosis.

Methods

The core of the paper is the GOALS operator. Instead of looking at model weights or complex game-theoretic subsets, GOALS uses a "sensitivity" approach:

  • The Perturbation: For a feature of interest, the method applies a small constant shift ($\xi$) to that feature across the dataset.
  • The Score: It measures the difference between the original model output ($f$) and the output after the shift ($g^{(j)}$). This difference, $\delta^{(j)} = f - g^{(j)}$, represents the local importance for every individual.
  • Global Aggregation: The average of these local differences across the population serves as the global importance score.

What makes it new? While sensitivity analysis isn't new, the authors derive the exact probabilistic properties for GOALS within Gaussian Process regression. They also introduce a "rank-one update" mathematical trick for kernel matrices, allowing GOALS to be computed in seconds for high-dimensional data where other methods (like RATE) might take hours. Finally, they demonstrate how this framework can be extended to Probabilistic Neural Networks.

Datasets

The authors validated GOALS using:

  • Synthetic Simulations: Six scenarios ranging from simple additive effects to complex pairwise interactions and population-specific effects (where a feature only matters for a subset of the group).
  • High-Dimensional Genetic Data: Real genotypes from the Wellcome Trust Case Control Consortium (WTCCC), involving ~3,000 samples and over 450,000 genetic features.
  • Heterogeneous Stock of Mice: A dataset of 1,814 mice with 10,227 genetic markers, used to map traits like body weight, CD8+ cell percentage, and HDL (cholesterol) levels.

Key Insights and Ideas

  • Function Space vs. Parameter Space: Most interpretability methods look at internal model parameters. GOALS looks at the function space—how the actual predictions change. This makes it more robust to how the model is structured internally.
  • Capturing "Hidden" Interactions: Traditional methods often project nonlinear models onto a linear space to explain them. The authors show that this "linear projection" often misses variables that only contribute to the outcome through interactions. GOALS catches these because it operates directly on the nonlinear function.
  • Multimodality is Information: In Scenario VI, the local GOALS scores for a specific feature showed a bimodal distribution. This "aha moment" revealed that the feature had a strong effect on one half of the population and zero effect on the other—a nuance that a purely global score would have averaged out.

Key Results

  • Superior Detection: In simulations with interaction effects, GOALS significantly outperformed LASSO, Random Forests, and BART in identifying causal variables (higher Area Under the ROC Curve).
  • Computational Speed: For a dataset with 500 features, the global method RATE took ~60 seconds, while GOALS took only one second. This scalability is vital for modern genomic studies.
  • Biological Discovery: When applied to mouse HDL levels, GOALS identified 13 relevant genes across five chromosomes that were missed by other state-of-the-art methods. It also successfully identified "cage effects," showing how local importance scores can capture environmental influences on genetic expression.

Limitations and Future Work

  • Disentanglement: While GOALS tells you that a variable is important, it doesn't explicitly distinguish between its linear contribution and its nonlinear interaction contribution. The authors suggest this as a primary path for future research.
  • Significance Thresholds: The current framework doesn't provide a formal "p-value." Users must rely on heuristic thresholds or permutation tests, which can be computationally expensive.
  • Hyperparameter Selection: The method requires choosing a perturbation constant ($\xi$). While the paper shows results are relatively robust to this choice, a standardized way to select the "optimal" $\xi$ is not yet established.

Summary updated

Abstract

The ability to interpret machine learning models has become increasingly important as their usage in data science continues to rise. Most current interpretability methods are optimized to work on either (i) a global scale, where the goal is to rank features based on their contributions to overall variation in an observed population, or (ii) the local level, which aims to detail on how important a feature is to a particular individual in the data set. In this work, a new operator is proposed called the “GlObal And Local Score” (GOALS): a simple post hoc approach to simultaneously assess local and global feature variable importance in nonlinear models. Motivated by problems in biomedicine, the approach is demonstrated using Gaussian process regression where the task of understanding how genetic markers are associated with disease progression both within individuals and across populations is of high interest. Detailed simulations and real data analyses illustrate the flexible and efficient utility of GOALS over state-of-the-art variable importance strategies.
BibTeX
@inproceedings{winnnunez2024simple,
  author = {Emily T. Winn-Nuñez and Maryclare Griffin and Lorin Crawford},
  title = {{A simple approach for local and global variable importance in nonlinear regression models}},
  booktitle = {Computational},
  year = {2024},
  doi = {10.1016/j.csda.2023.107914},
}

Loading related content...

Figure