Standard Gaussian Process (GP) regression typically assumes that input features are independent, using a diagonal weight matrix to determine feature relevance. However, this assumption often falls short in complex, high-dimensional datasets where the true underlying drivers are combinations of features. By utilizing a full covariance matrix, GPs can automatically discover "hidden features"—linear combinations of inputs that actually drive the underlying process—significantly improving prediction accuracy.
The Blind Spot of Automatic Relevance Determination
In supervised learning, we often assume the target variable $y$ depends directly on the observed input features $\mathbf{x}$. A common technique in Gaussian Process regression is Automatic Relevance Determination (ARD), which assigns a weight (a "length-scale") to each input dimension. If a dimension has a small weight, it is deemed irrelevant.
However, ARD has a major blind spot: it only evaluates features aligned with the coordinate axes. In many real-world problems, the "true" features are actually linear combinations of the inputs (e.g., $z = x_1 + x_2$). Standard ARD treats $x_1$ and $x_2$ as separate entities and fails to capture the interaction between them. To bridge this gap, researchers allow the GP to "rotate" its view of the input space to find these hidden directions.
Methodology: The Full Covariance Matrix
To capture these interactions, the standard squared-exponential covariance function is modified:
$$C_p(\mathbf{x}, \mathbf{x'}) = \sigma_p^2 \exp \left[ -\frac{1}{2}(\mathbf{x} - \mathbf{x'})^T W (\mathbf{x} - \mathbf{x'}) \right]$$
- The Innovation: Instead of restricting $W$ to be a diagonal matrix (as in ARD), $W$ is allowed to be a general positive semidefinite matrix.
- Parameterization: To ensure $W$ remains valid (positive semidefinite) during optimization, a Cholesky decomposition ($W = U^T U$) is used, where $U$ is an upper triangular matrix.
- Training: The parameters (the elements of $U$, the signal variance $\sigma_p^2$, and the noise variance $\sigma_\nu^2$) are learned by maximizing the log-likelihood of the training data using a conjugate gradient algorithm.
- Feature Discovery: After training, an eigen-analysis of $W$ is performed. The eigenvectors associated with large eigenvalues represent the "hidden features" (the directions in input space that matter most), while the number of significant eigenvalues reveals the true dimensionality of the hidden feature space.
By using a full matrix $W$, the GP effectively learns a custom Mahalanobis distance. This is mathematically equivalent to performing a linear projection of the data into a lower-dimensional "hidden" space before applying the GP.
Experimental Validation
The approach was evaluated on synthetic regression tasks designed to test feature discovery, including a 1D trigonometric function embedded in a 2D space, and a high-interaction surface benchmark (from Breiman, 1993). The latter target is a sum of three sigmoids, where each sigmoid depends on a different linear combination of 10 input variables ($x \in \mathbb{R}^{10}$). This tests the model's ability to find a 3D subspace within a 10D input space.
Key Results
- Superior Accuracy: On the trigonometric task, the full matrix ($W_f$) significantly outperformed the diagonal matrix ($W_d$). For datasets larger than 32 points, the relative error improvement was often greater than 75%.
- Successful Subspace Discovery: In the 10D Breiman task, the eigen-analysis of $W_f$ correctly identified exactly three dominant eigenvalues. The corresponding eigenvectors spanned the same subspace as the true hidden features used to generate the data.
- Failure of ARD: As seen in the figure above, the diagonal matrix $W_d$ assigned roughly equal importance to all 10 inputs in the Breiman task, failing to realize that the function only varied along three specific directions.
Limitations and Future Directions
While powerful, a full matrix $W$ has $d(d+1)/2$ parameters, whereas a diagonal matrix has only $d$. This extra flexibility allows for better modeling of interactions but requires more training data to avoid overfitting. For very small datasets, the diagonal model sometimes performs better because the full model overfits the noise.
Future work could explore a full Bayesian approach (integrating over the parameters of $W$) to mitigate overfitting issues seen with small samples. Additionally, explicitly restricting the rank of $W$ to a value $q < d$ could force the model to find a $q$-dimensional bottleneck, potentially acting as a strong form of regularization for high-dimensional problems.
Summary updated