Abstract
We study a distributionally robust optimization formulation (i.e., a min-max game) for two representative problems in Bayesian nonparametric estimation: Gaussian process regression and, more generally, linear inverse problems. Our formulation seeks the best mean-squared error predictor, in an infinite-dimensional space, against an adversary who chooses the worst-case model in a Wasserstein ball around a nominal infinite-dimensional Bayesian model. The transport cost is chosen to control features such as the degree of roughness of the sample paths that the adversary is allowed to inject. We show that the game has a well-defined value (i.e., strong duality holds in the sense that max-min equals min-max) and that there exists a unique Nash equilibrium which can be computed by a sequence of finite-dimensional approximations. Crucially, the worst-case distribution is itself Gaussian. We explore properties of the Nash equilibrium and the effects of hyperparameters through a set of numerical experiments, demonstrating the versatility of our modeling framework.
Introduction
In Bayesian nonparametric estimation, Gaussian Processes (GPs) serve as a foundational tool for modeling unknown functions. Despite their flexibility, GPs are notoriously sensitive to model misspecification. If the initial assumptions regarding the function's smoothness (the "nominal" prior) or the noise characteristics are even slightly inaccurate, the resulting predictions and uncertainty estimates can be significantly compromised.
While Distributionally Robust Optimization (DRO) has been widely applied to finite-dimensional data to mitigate such sensitivities, extending these principles to the infinite-dimensional setting—where the goal is to estimate an entire function—presents significant theoretical and computational challenges. This paper introduces a framework for robust GP regression and linear inverse problems by framing the estimation as a min-max game against an adversary who can perturb the model within a Wasserstein ball.
The Distributionally Robust Framework
The core of this research is a zero-sum game between two players: a Decision-Maker seeking to minimize the Mean-Squared Error (MSE) and an Adversary seeking to maximize it. The Adversary operates within an ambiguity set, defined by a Wasserstein distance from a nominal Gaussian model.
Wasserstein Ambiguity and RKHS Ground Cost
To define distance in an infinite-dimensional space, the authors utilize a norm derived from a Reproducing Kernel Hilbert Space (RKHS). This choice of "ground cost" is critical; it allows the modeler to specify exactly which features the adversary can manipulate. By adjusting the weights within the RKHS norm, one can control the degree of "roughness" or high-frequency noise the adversary is permitted to inject into the sample paths. This ensures that the robustness is targeted toward the most likely forms of model misspecification.
Strong Duality and the Gaussian Adversary
A central theoretical contribution of this work is the proof of strong duality. The authors demonstrate that the order of the min-max game does not affect the outcome, guaranteeing the existence of a stable Nash equilibrium.
Perhaps most surprisingly, the authors prove that the worst-case distribution chosen by the adversary is itself Gaussian. This result is of immense practical value: it ensures that the robust estimator remains an affine function of the observations. Consequently, the robust framework retains the computational familiarity and efficiency of standard Gaussian Process regression while providing a much-needed layer of protection against model errors.
Numerical Validation and Inverse Problems
The framework's versatility is demonstrated through several numerical experiments involving both standard regression and complex linear inverse problems. These include:
- Gaussian Process Regression: Testing on the unit interval $[0, 1]$ to observe how the robust model handles varying degrees of prior uncertainty.
- Linear Inverse Problems: The authors apply the method to the Laplace equation (source term recovery), the Heat equation (initial condition recovery), and the Radon transform (a mathematical foundation for medical imaging like CT scans).
In these experiments, the robust model exhibits "honest uncertainty." Unlike nominal models that may be overconfident in regions with sparse data, the robust estimator inflates the variance, effectively signaling that the prior assumptions may not hold in those regions. The robust posterior often displays "ripples" or sharper contrasts, indicating a strategic hedging against potential correlations that a standard model would ignore.
Computational Implementation
To solve the infinite-dimensional problem, the authors propose a sequence of finite-dimensional approximations. These approximations are shown to converge to the true solution and can be solved efficiently using a tailored version of the Frank-Wolfe algorithm. This approach allows the robust estimator to be computed using existing Semi-Definite Programming (SDP) solvers, making the framework accessible for practical applications.
Conclusion and Future Directions
This paper provides a rigorous foundation for distributionally robust Bayesian estimation in infinite dimensions. By proving that the worst-case scenario remains Gaussian and establishing strong duality, the authors bridge the gap between robust optimization and nonparametric Bayesian statistics.
Future work remains in several areas:
- Hyperparameter Selection: Developing data-driven methods (such as cross-validation) to select the Wasserstein radius $\delta$ and RKHS weights $w_n$.
- Scaling: Improving the efficiency of finite-dimensional approximations for high-dimensional or highly complex fields.
- Non-Gaussian Extensions: Exploring whether these robust principles can be applied to non-Gaussian nominal models, such as heavy-tailed distributions, which are common in robust statistics but difficult to handle in infinite-dimensional settings.
Summary updated