Distributionally Robust Optimization with Maximum Mean Discrepancy

12 concepts8 papers1 proposal

When deploying machine learning models or control systems in the real world, we rarely encounter the exact data distribution seen during training. Distributionally Robust Optimization (DRO) addresses this vulnerability by optimizing performance not just for a single empirical distribution, but for the worst-case distribution within a surrounding neighborhood, known as an ambiguity set.

Traditionally, these ambiguity sets are constructed using statistical distances like the Wasserstein metric or $f$-divergences. However, these classical approaches face a severe bottleneck: the curse of dimensionality. As the number of data dimensions grows, the amount of data required to reliably estimate a Wasserstein neighborhood grows exponentially.

To overcome this limitation, a powerful paradigm has emerged: defining ambiguity sets using Kernel Mean Embeddings (KME) and the Maximum Mean Discrepancy (MMD). By mapping probability distributions into a high-dimensional Reproducing Kernel Hilbert Space (RKHS), this framework provides a dimension-free, computationally tractable approach to robust decision-making.

The Intuition: Geometry in the Hilbert Space

To understand why kernel-based DRO works, we must look at how it redefines the distance between probability distributions.

Instead of comparing distributions by shifting probability mass across a coordinate space (as in Wasserstein distance), a kernel mean embedding represents an entire probability distribution as a single point in an RKHS. The distance between two distributions in this space is their Maximum Mean Discrepancy (MMD).

As shown by Jia‐Jie Zhu et al. in Worst-Case Risk Quantification under Distributional Ambiguity using Kernel Mean Embedding in Moment Problem, this embedding transforms an abstract, infinite-dimensional space of probability measures into a clean geometric structure. An ambiguity set can be defined simply as a ball in the RKHS centered on the empirical data embedding:

$$\mathcal{P} = \{ P : \|\mu_P - \mu_{\hat{P}}\|_{\mathcal{H}} \le \rho \}$$

where $\mu_P$ is the kernel mean embedding of distribution $P$, and $\rho$ is the uncertainty radius. Licio Romao et al. leverage this geometric formulation in Distributionally Robust Optimal and Safe Control of Stochastic Systems via Kernel Conditional Mean Embedding to tractably search for worst-case distributions, enabling safe control even when system dynamics are learned from noisy, limited data.

Escaping the Curse of Dimensionality

The primary reason to prefer MMD over other metrics is its convergence behavior in high-dimensional spaces.

In a Wasserstein-based framework, the empirical distribution converges to the true distribution at a rate of $O(n^{-1/d})$, where $d$ is the data dimension. If your data has dozens or hundreds of features, the ambiguity set must be massive just to guarantee it contains the true distribution, rendering the optimization overly conservative or useless.

In contrast, MMD converges at a rate of $O(n^{-1/2})$, completely independent of the input dimension $d$. As proven by Matthew Staib and Stefanie Jegelka in Distributionally Robust Optimization and Generalization in Kernel Methods, this dimension-free property means that the size of the ambiguity set does not need to blow up as data complexity increases.

This dimension-independent scaling unlocks several practical applications:

Duality and the Robust Representer Theorem

Searching over an infinite-dimensional space of arbitrary probability distributions to find the "worst-case" scenario sounds computationally impossible. The magic of kernel-based DRO lies in how this minimax problem simplifies.

In Kernel Distributionally Robust Optimization: Generalized Duality Theorem and Stochastic Approximation, Jia-Jie Zhu et al. derive a generalized duality theorem for this framework. They show that the inner maximization over probability distributions can be rewritten as a manageable dual problem. Crucially, this duality does not require restrictive assumptions on the loss function—such as a known Lipschitz constant or polynomial form—making it compatible with modern, non-convex models like neural networks via stochastic functional gradient algorithms.

Furthermore, kernel methods bring a powerful tool to the table: the Robust Representer Theorem. As explained by Yassine Nemmour et al. in Maximum Mean Discrepancy Distributionally Robust Nonlinear Chance-Constrained Optimization with Finite-Sample Guarantee, embedding the uncertainty into an RKHS allows complex, non-convex, and nonlinear constraints to be solved using standard numerical solvers without requiring linear approximations.

This mathematical tractability bridges the gap between robust statistical theory and real-time engineering. For instance, Bernhard Schölkopf et al. demonstrate in Approximate Distributionally Robust Nonlinear Optimization with Application to Model Predictive Control: A Functional Approach that MMD-based DRO can be directly integrated into Model Predictive Control (MPC) systems, guaranteeing safety and constraint satisfaction even under sudden environmental shifts.

Key Foundational Work

Distributionally Robust Optimization and Generalization in Kernel Methods (2019) — contributed 2 concept elements to this direction.
Worst-Case Risk Quantification under Distributional Ambiguity using Kernel Mean Embedding in Moment Problem (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_kernel_based_uncertainty_sets_ec705ff0,
  title = {kernel-based uncertainty sets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kernel-based-uncertainty-sets/rdr_ec705ff09cbf484a8b49dcf69d2c98f0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_kernel_based_uncertainty_sets_ec705ff0_v1,
  title = {kernel-based uncertainty sets},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/kernel-based-uncertainty-sets/rdr_ec705ff09cbf484a8b49dcf69d2c98f0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake