Learning Feature Importance through Kernel Lengthscales

37 concepts28 papers1 proposal

When building predictive models, high-dimensional datasets frequently present a challenge: many input features are simply noise or irrelevant to the target task. In parametric linear models, feature selection is straightforward. We can inspect the magnitude of learned coefficients or apply regularization techniques like the Lasso to drive irrelevant weights directly to zero.

However, non-parametric kernel methods—such as Gaussian Processes (GPs) and Support Vector Machines (SVMs)—do not use explicit linear coefficients. Instead, they operate by mapping inputs into an implicit, often infinite-dimensional feature space via a kernel function. As noted by Matthias Seeger et al. in Bayesian Model Selection for Support Vector Machines, Gaussian Processes and Other Kernel Classifiers, traditional feature selection methods like manual grid searches become computationally impossible as the number of dimensions grows.

Automatic Relevance Determination (ARD) solves this problem. It embeds feature selection directly into the kernel's coordinate system, allowing the model to automatically isolate and ignore irrelevant variables during training.

From Isotropic to Anisotropic Kernels

To understand how ARD works, consider a standard, isotropic Squared Exponential (or Radial Basis Function) kernel. This function measures the similarity between two data points, $x$ and $x'$, based on their Euclidean distance:

$$k(x, x') = \sigma^2 \exp \left( -\frac{\|x - x'\|^2}{2\ell^2} \right) = \sigma^2 \exp \left( -\frac{1}{2\ell^2} \sum_{i=1}^D (x_i - x'_i)^2 \right)$$

Here, $\ell$ is a single global hyperparameter known as the lengthscale. It dictates how far you must move in the input space before the function values become uncorrelated. Because there is only one lengthscale for all $D$ dimensions, this kernel assumes that a one-unit change along feature $A$ impacts similarity exactly the same as a one-unit change along feature $B$. If feature $B$ is pure noise, the model will still let variations along that axis distort its predictions.

ARD alters this geometry by assigning an independent lengthscale $\ell_i$ to each input dimension $i$. This transforms the isotropic kernel into an anisotropic kernel:

$$k(x, x') = \sigma^2 \exp \left( -\frac{1}{2} \sum_{i=1}^D \frac{(x_i - x'_i)^2}{\ell_i^2} \right)$$

As Johan Suykens et al. demonstrate in Automatic relevance determination for Least Squares Support Vector Machines classifiers, this formulation embeds a diagonal weighting matrix directly into the kernel. By scaling each input dimension independently before computing the inner product, the importance of individual features becomes explicit, even when the underlying feature mapping remains implicit.

The Geometry of Infinite Lengthscales

The reason ARD works so effectively lies in the optimization dynamics of these individual lengthscales. Typically, kernel hyperparameters are tuned by maximizing the marginal likelihood (or evidence) of the model.

Imagine that the $i$-th feature in your dataset is completely irrelevant to the target variable. If the model attempts to fit variations along this axis, it will overfit the noise, which severely penalizes the marginal likelihood. To maximize the evidence, the optimization routine will drive that specific lengthscale $\ell_i$ toward infinity ($\ell_i \to \infty$).

Let's look at what happens to the mathematical expression as a lengthscale grows:

$$\lim_{\ell_i \to \infty} \frac{(x_i - x'_i)^2}{\ell_i^2} = 0$$

When $\ell_i$ approaches infinity, the contribution of the $i$-th feature to the summation drops to zero. Consequently, no matter how much two data points differ along that dimension, the kernel treats their distance along that axis as zero. The feature is rendered completely invisible to the model.

Conversely, highly informative features receive small lengthscales ($\ell_i$). A small lengthscale means that even a tiny change along that dimension will cause the kernel similarity to drop rapidly, signaling to the model that this feature is highly influential. As highlighted by Zeel B. Patel et al. in Accurate and Scalable Gaussian Processes for Fine-Grained Air Quality Inference and Suresh Bishnoi et al. in Predicting Young's modulus of oxide glasses with sparse datasets using machine learning, this behavior transforms hyperparameter tuning into an embedded feature selection mechanism. The data itself stretches and compresses the input space, automatically pruning away uninformative dimensions.

Mathematical Justification and Interpretability

Why can we confidently trust these inverse lengthscales as a proxy for feature importance? Vu Viet Hoang et al. formalize this relationship in High Dimensional Bayesian Optimization using Lasso Variable Selection by linking inverse lengthscales ($1/\ell_i$) directly to the partial derivatives of the underlying target function. Dimensions with high sensitivity require small lengthscales to track rapid changes, while insensitive dimensions naturally receive large lengthscales.

This mathematical link provides a powerful window into otherwise opaque models. For example, Evgeny Burnaev et al. show in Forecasting of Commercial Sales with Large Scale Gaussian Processes that ARD kernels offer strategic business insights by automatically ranking which external factors drive retail sales. Similarly, in Predicting and Characterising User Impact on Twitter, Νικόλαος Αλέτρας leverages ARD to rank behavioral and textual features, providing clear interpretability without sacrificing the non-linear flexibility of Gaussian Processes.

Extending ARD to Diverse Domains

While the intuition is easiest to visualize with continuous Euclidean distances, the core principle of ARD generalizes across a wide variety of data structures and problem settings:

By allowing the data to dynamically reshape the geometry of the kernel space, ARD bridges the gap between non-parametric flexibility and structural interpretability, protecting kernel methods from the curse of dimensionality.

Key Foundational Work

Bayesian Model Selection for Support Vector Machines, Gaussian Processes and Other Kernel Classifiers (1999) — contributed 2 concept elements to this direction.
Automatic relevance determination for Least Squares Support Vector Machines classifiers (2001) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automatic_relevance_determination_kernel_1c85aa6f,
  title = {automatic relevance determination (kernel-based feature selection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automatic-relevance-determination-kernel-based-feature-selection/rdr_1c85aa6fde6d49c19a35ca73be4c3cba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_automatic_relevance_determination_kernel_1c85aa6f_v1,
  title = {automatic relevance determination (kernel-based feature selection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automatic-relevance-determination-kernel-based-feature-selection/rdr_1c85aa6fde6d49c19a35ca73be4c3cba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake