Abstract
Derivatives are a key nonparametric functional in wide-ranging applications where the rate of change of an unknown function is of interest. In the Bayesian paradigm, Gaussian processes (GPs) are routinely used as a flexible prior for unknown functions. However, the "plug-in" strategy—differentiating the posterior distribution of a GP to estimate derivatives—has been previously perceived as suboptimal. This article provides the first theoretical proof that plug-in GPs actually achieve minimax optimal contraction rates simultaneously for all derivative orders. By establishing that a single data-driven hyperparameter, tuned via Empirical Bayes, works for both the function and its derivatives, this work provides a practically simple and theoretically grounded method for derivative estimation. Simulations and an application to global sea-level rise data demonstrate the competitive performance of this approach.
Introduction
In fields ranging from meteorology to computer vision, the rate of change of a process is often more critical than the absolute values of the process itself. For example, in climate science, understanding the acceleration of sea-level rise is vital for coastal planning. While Gaussian Processes (GPs) are the "gold standard" for nonparametric Bayesian regression, their use for derivative estimation has historically been controversial.
For over a decade, the prevailing wisdom suggested that the "plug-in" approach—simply differentiating the GP posterior—was "degraded" or suboptimal. The heuristic belief was that if a GP was tuned to fit the underlying function accurately, the derivative estimates would suffer, and vice versa. This led to the development of complex, order-specific estimation methods. This paper sets out to determine if this perceived suboptimality is a mathematical reality or a misunderstanding of GP theory.
The Plug-in Strategy and Theoretical Framework
The authors analyze the plug-in strategy, which involves assigning a GP prior to a regression function $f$ and then using the derivatives of the resulting posterior distribution to infer the $k$-th derivative $f^{(k)}$.
Posterior Contraction Rates
The core of the paper's theoretical contribution is the derivation of posterior contraction rates. These rates describe the speed at which the posterior distribution "shrinks" toward the true derivative as the sample size $n$ increases. Using an operator-theoretic framework—specifically Mercer’s theorem and the concept of "equivalent kernels"—the authors characterize how the GP behaves in Reproducing Kernel Hilbert Spaces (RKHS).
Simultaneous Optimality
A major discovery is the nonparametric plug-in property. Unlike local polynomials or smoothing splines, where the optimal tuning parameter depends on the specific derivative order being estimated, a GP possesses a unique adaptability. The authors prove that the same choice of the regularization parameter $\lambda$ allows the GP to achieve the minimax optimal rate for the function and all its derivatives simultaneously. This property ensures that a model optimized for the primary function is automatically "good enough" for its derivatives.
Data-Driven Hyperparameter Tuning
To make the plug-in GP practical, the authors propose using Empirical Bayes (EB) to select hyperparameters, such as the noise variance $\sigma^2$ and the smoothing parameter $\lambda$. By maximizing the marginal likelihood, the EB approach selects parameters that satisfy the optimal rate condition while maintaining computational efficiency.
This finding debunks the idea of inherent suboptimality; the authors suggest that previous "degradation" observed in literature was likely the result of poor hyperparameter tuning rather than a flaw in the GP framework itself. Furthermore, the EB-tuned plug-in GP is robust to "oversmoothing," meaning that even if a highly smooth kernel (like the Squared Exponential kernel) is used on a less-smooth function, the model can still achieve optimal rates.
Empirical Validation
Simulation Studies
The paper validates these theoretical findings through synthetic experiments. Using functions with Hölder smoothness $\alpha=3$, the authors compared plug-in GPs (using Matérn and Sobolev kernels) against B-spline random series priors across sample sizes of $n=100, 500, \text{ and } 1000$. The results consistently showed that plug-in GPs produced lower Root Mean Square Errors (RMSE) for derivative estimation than the B-spline alternatives.
Global Mean Sea-Level (GMSL) Analysis
The method was applied to a real-world dataset of tide-gauge records from 1880 to 2009 ($n=130$). The goal was to estimate the rate of sea-level rise over time. The plug-in GP successfully captured the non-linear acceleration of sea-level rise, estimating a change from approximately 1.22 mm/yr in 1880 to 1.81 mm/yr in 2010. This provides a more flexible and theoretically grounded estimate than simple linear regression models often used in climate reports.
Conclusion and Future Work
This work provides the first positive theoretical result for plug-in GPs in the context of derivative functionals. It establishes that the simplest method for derivative estimation is, in fact, minimax optimal.
While the paper focuses on 1D cases with random designs, it opens several avenues for future research:
- Fixed Designs: Extending the theory to cases where inputs are pre-determined rather than sampled from a distribution.
- Multivariate Extensions: Explicitly mapping out the theoretical rates for high-dimensional partial derivatives.
- Norm Refinement: Further refining the $L_\infty$ (supremum) norm analysis for derivatives, which remains a complex area of proof.
By providing a computationally simple and theoretically robust framework, this research rehabilitates the plug-in GP as a primary tool for scientists and engineers interested in the dynamics of change.
Summary updated