In traditional statistics, we often summarize a complex dataset using simple metrics like the mean or variance. While useful, these summaries are "lossy"—they discard the rich geometry of the underlying probability distribution. To capture the full picture, researchers have historically relied on parametric models (assuming the data is Gaussian, for example) or density estimation. However, these methods struggle in high dimensions or with non-standard noise.
A powerful alternative has emerged: representing an entire probability distribution as a single point in a high-dimensional feature space. This framework, known as Kernel Mean Embeddings (KME), allows us to treat complex distributions as simple vectors in a Reproducing Kernel Hilbert Space (RKHS). By doing so, we can perform sophisticated statistical inference using the tools of linear algebra.
The Distribution as a Single Point
The core intuition behind a kernel mean embedding is to "fingerprint" a distribution by taking the expectation of a feature map. If we have a distribution $P$ and a feature map $\phi(x)$ that maps data into a high-dimensional space, the mean embedding $\mu_P$ is simply the average position of the data in that space:
$$\mu_P = \mathbb{E}_{X \sim P} [\phi(X)]$$
As Bharath K. Sriperumbudur, Arthur Gretton, and Kenji Fukumizu established in Hilbert space embeddings and metrics on probability measures, this mapping is injective (one-to-one) if we use a "characteristic kernel."
When a kernel is characteristic—like the popular Gaussian RBF kernel—the mean embedding $\mu_P$ captures every single moment of the distribution. It doesn't just store the mean; it implicitly encodes the variance, skewness, kurtosis, and every higher-order statistic. This allows us to distinguish between two distributions even if they share the same average value, a property Clement Benard and colleagues leverage to identify variable importance in random forests based on distributional shifts rather than just mean shifts.
Measuring Distance with Maximum Mean Discrepancy
Once distributions are represented as points in a Hilbert space, comparing them becomes a simple matter of measuring the distance between those points. This distance is called the Maximum Mean Discrepancy (MMD).
MMD provides a "distribution-free" way to determine if two sets of samples come from the same source. In M3D: a kernel-based test for shape changes in methylation profiles, Gabriele Schweikert and Guido Sanguinetti use MMD to detect subtle changes in the "shape" of DNA methylation, which traditional tests might miss.
The beauty of MMD lies in its computational efficiency. Because of the "kernel trick," we never actually have to compute the high-dimensional coordinates of $\phi(x)$. We only need to compute the kernel function between pairs of samples. This makes MMD highly effective for high-dimensional data, such as evaluating the quality of images generated by GANs, as shown by Gao Huang and Chuan Guo in their empirical study of GAN metrics.
Conditional Distributions as Operators
The framework extends beyond static distributions. In Hilbert space embeddings of conditional distributions with applications to dynamical systems, Le Song and Alex Smola introduced Conditional Mean Embeddings (CME).
Instead of representing a distribution as a vector, a CME represents a conditional distribution $P(Y|X)$ as a linear operator. This operator maps the embedding of a specific observation $x$ to the embedding of the corresponding conditional distribution of $y$. This transformation turns complex probabilistic reasoning—like the filtering steps in a hidden Markov model—into simple matrix multiplications. This is particularly useful in random dynamical systems, where Yoshinobu Kawahara and others use operator-valued kernels to embed stochastic transitions directly into the functional space.
Applications in Causal Inference and Adaptation
The ability to compare full distributions has profound implications for causal inference. Traditional methods often look for the "Average Treatment Effect" (ATE). However, a drug might not change the average outcome but might significantly increase the variance (helping some, hurting others).
Jake Fawkes and Dino Sejdinovic address this in Doubly Robust Kernel Statistics for Testing Distributional Treatment Effects, using MMD to detect changes in the entire shape of the outcome distribution. Similarly, in Active Estimation of Causal Quantities, Erdun Gao uses CMEs to represent target distributions, allowing for sample-efficient causal discovery without explicit density estimation.
This "distributional matching" is also the engine behind modern domain adaptation. Jianmin Wang and Michael I. Jordan developed Joint Adaptation Networks that use a Joint MMD criterion to align the distributions of features and labels across different domains. By minimizing the distance between these embeddings, models can generalize from a source domain (like synthetic images) to a target domain (like real-world photos) even when the underlying distributions shift significantly.