Abstract
Standard generative models like Restricted Boltzmann Machines (RBMs) are typically trained by maximizing likelihood, which minimizes the Kullback-Leibler (KL) divergence. However, this objective is "geometrically blind," ignoring the underlying distance between data points. We present a training framework that utilizes the Wasserstein distance (Earth Mover’s Distance) to allow RBMs to learn distributions that respect the data's geometry. By employing entropic regularization and the Sinkhorn algorithm, we derive a practical gradient for RBM parameters. Our results on MNIST and UCI PLANTS datasets demonstrate that Wasserstein-trained RBMs produce cleaner samples and achieve superior performance in denoising and data completion tasks.
Introduction: The Geometric Blindness of KL Divergence
The traditional "gold standard" for training generative models is the minimization of Kullback-Leibler (KL) divergence between the model distribution $p_\theta$ and the empirical data distribution $\hat{p}$. While mathematically convenient, KL divergence fails to account for the metric space in which the data resides. For instance, in image generation, a model's mistake is treated identically by the KL objective regardless of whether the generated image is visually similar to the target or completely unrecognizable noise, provided the exact bit-patterns do not match.
As illustrated in the figure above, a standard RBM trained with KL divergence might place probability mass on a state that is bit-wise distinct but geographically distant from the true data, simply to satisfy the likelihood objective. In contrast, a Wasserstein-based approach encourages the model to place probability mass near the data according to a meaningful ground metric, such as Hamming distance.
Methodology: Learning with the Wasserstein Distance
The core innovation of this work is replacing the KL divergence objective with the Wasserstein distance. The Wasserstein distance measures the minimum "work" required to transform one probability distribution into another, given a ground metric.
Entropic Regularization and the Sinkhorn Algorithm
Computing the exact Wasserstein distance for the $2^d$ states of an RBM is computationally prohibitive. To overcome this, we utilize the $\gamma$-smoothed Wasserstein distance, which introduces an entropy term to the transport plan. This regularization allows for the use of the Sinkhorn algorithm—a fast, iterative method to compute the distance and its gradients efficiently.
Gradient Derivation
We derive a practical gradient for the RBM's free energy parameters. This gradient involves "dual potentials" ($\alpha^*$), which function as a weighting mechanism. These potentials identify generated samples that are "expensive" (i.e., far from real data) and signal the model to lower their probability. To ensure stability, we employ a hybrid objective that includes a KL regularization term and weight decay, effectively fine-tuning a standard RBM using the Wasserstein objective.
Experimental Results
The proposed framework was evaluated on two binary datasets: binarized MNIST digits (downsampled to 14x14) and the UCI PLANTS dataset (70-dimensional binary vectors).
Qualitative and Quantitative Performance
Samples generated by the Wasserstein-trained RBM (RBM-W) are visually "cleaner" and more well-defined than those from standard RBMs. For the MNIST "0" class, the RBM-W produces sharper contours and more consistent shapes. Quantitatively, the RBM-W outperformed standard RBMs and Kernel Density Estimation (KDE) in data completion and denoising tasks, specifically in terms of expected Hamming error.
Prototypical Clustering
A key insight from our analysis is that RBM-W models tend to learn a finite set of "clean" prototypical clusters rather than attempting to cover the entire data manifold uniformly. PCA visualizations confirm that while standard RBMs spread probability mass thinly, RBM-W concentrates it into dense, meaningful islands corresponding to the core structure of the data. This behavior represents a unique bias-variance tradeoff: the model exhibits higher bias by pulling noisy data toward its learned prototypes but significantly lower variance, making it highly robust to heavy noise.
Conclusion and Future Work
Wasserstein training provides a principled way to incorporate geometric information into Boltzmann machines. While the approach leads to more robust and visually coherent models, it faces challenges such as potential loss of diversity (mode collapse) and increased computational overhead compared to standard Contrastive Divergence. Future research will explore scaling this approach to deeper architectures like Deep Boltzmann Machines and integrating more flexible noise models to mitigate diversity loss.
Summary updated