Identifiability for Gaussian Processes with Holomorphic Kernels

Ameer Qaqish, Didong LiICLR 2025 · OpenReview

When a Gaussian Process (GP) model outputs "lengthscale" or "variance" parameters, can those values be trusted as unique, meaningful representations of the underlying data? This paper introduces a novel mathematical framework to prove that parameters in popular smooth kernels—such as the Squared Exponential and Periodic kernels—are indeed identifiable, meaning they can be uniquely recovered given sufficient data.

The Challenge of Parameter Identifiability

In Gaussian Process modeling, researchers often care about more than just predictive accuracy; the kernel parameters themselves carry vital domain interpretations. For instance, in spatial transcriptomics, a specific "lengthscale" might represent the physical distance over which gene expression is correlated.

However, interpreting these parameters carries a hidden danger: identifiability. If a parameter is non-identifiable, different sets of parameter values can produce the exact same statistical distribution. If a model cannot distinguish between a "high variance with a short lengthscale" and a "low variance with a long lengthscale," any domain-specific interpretation of those numbers is essentially guesswork.

While parameter identifiability has been well-studied for the Matérn kernel family (where, surprisingly, variance and lengthscale are often not individually identifiable), the mathematics for the most popular "smooth" kernels used in machine learning—like the Squared Exponential (SE) or Periodic kernels—have remained a significant gap in the literature. Because these kernels are "too smooth" for traditional mathematical tools like the integral test, researchers have been left unsure if their parameter interpretations are theoretically sound.

A New Framework for Holomorphic Kernels

The authors bridge this gap by focusing on holomorphic kernels—functions that are not just infinitely differentiable, but can be analytically extended into the complex plane.

Their key contribution is a general theorem that breaks a kernel down into two distinct parts:

  1. Continuous component: Kernels with a continuous spectral density (e.g., the Squared Exponential kernel).
  2. Discrete component: Kernels with a discrete spectrum (e.g., Periodic or Cosine kernels).

By analyzing these components separately, the authors developed a systematic pipeline to check for identifiability. Instead of relying on the decay of spectral density (which fails for infinitely smooth kernels), they leverage the properties of holomorphic functions to prove that if two such GPs are statistically equivalent, their underlying parameters must be identical.

Summary of evaluated kernels and their identifiable (microergodic) parameters.

Key Insights and Empirical Validation

To validate their theoretical proofs, the authors utilized simulated datasets generated from several kernel types, including Squared Exponential (SE), Periodic (Per), Damped Periodic (DPer), Rational Quadratic (RQ), and Cosine kernels. They observed whether Maximum Likelihood Estimators (MLEs) converged to the true parameter values as the number of data points increased.

  • Smoothness is a Strength: Unlike Matérn kernels, where high-frequency "wiggliness" makes it mathematically difficult to separate variance from lengthscale, the infinite smoothness of holomorphic kernels actually makes their parameters easier to identify.
  • The Mauna Loa Validation: The authors provide the first rigorous proof that all 11 parameters in the famous Mauna Loa $CO_2$ trend model (a complex composite kernel widely cited in GP literature) are identifiable, provided the noise lengthscale is smaller than the trend lengthscale. This formally justifies nearly two decades of researchers interpreting those specific parameters.
  • A Warning on Products: The framework also reveals limits to identifiability. While sums of periodic kernels are identifiable, products of more than three Cosine kernels can become non-identifiable. In these cases, different combinations of frequencies can result in the exact same kernel, making individual parameter interpretation impossible.
  • Empirical Consistency: As the sample size increased from 500 to 5,000 in simulations, the estimated parameters tightened significantly around the ground truth values. The MLEs for SE, Periodic, and RQ kernels showed no significant bias. Conversely, for the Cosine kernel, simulations confirmed the theoretical prediction that variance is not identifiable if the frequency is known.

Limitations and Future Work

While this paper establishes a robust theoretical foundation, several challenges remain:

  • MLE Optimization: Proving that parameters can be identified does not guarantee that standard Maximum Likelihood Estimation algorithms will easily find them. MLE loss surfaces can be highly multi-modal, making numerical optimization difficult in practice.
  • Stationarity: The current framework is limited to stationary kernels, where correlation depends only on the relative distance between points rather than their absolute locations. Extending these proofs to non-stationary GPs remains a major open challenge.
  • Non-Holomorphic Smooth Kernels: There exist infinitely differentiable kernels that are not holomorphic. The authors note that these edge cases require their own dedicated future study.

Summary updated

Abstract

Gaussian processes (GPs) are widely recognized for their robustness and flexibility across various domains, including machine learning, time series, spatial statistics, and biomedicine. In addition to their common usage in regression tasks, GP kernel parameters are frequently interpreted in various applications. For example, in spatial transcriptomics, estimated kernel parameters are used to identify spatial variable genes, which exhibit significant expression patterns across different tissue locations. However, before these parameters can be meaningfully interpreted, it is essential to establish their identifiability. Existing studies of GP parameter identifiability have focused primarily on Mat\'ern-type kernels, as their spectral densities allow for more established mathematical tools. In many real-world applications, particuarly in time series analysis, other kernels such as the squared exponential, periodic, and rational quadratic kernels, as well as their combinations, are also widely used. These kernels share the property of being holomorphic around zero, and their parameter identifiability remains underexplored. In this paper, we bridge this gap by developing a novel theoretical framework for determining kernel parameter identifiability for kernels holomorphic near zero. Our findings enable practitioners to determine which parameters are identifiable in both existing and newly constructed kernels, supporting application-specific interpretation of the identifiable parameters, and highlighting non-identifiable parameters that require careful interpretation.
BibTeX
@inproceedings{qaqish2025identifiability,
  author = {Ameer Qaqish and Didong Li},
  title = {{Identifiability for Gaussian Processes with Holomorphic Kernels}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=FUaDMRVrbS},
}
Figure