Informative GANs via Structured Regularization of Optimal Transport

Pierre Bréchet, Tao Wu, Thomas Möllenhoff, Daniel CremersarXiv 2019 · arXiv

Introduction

A fundamental challenge in unsupervised learning is disentangled representation learning: training models where individual latent dimensions correspond to specific, human-interpretable features of the data, such as the identity, rotation, or thickness of a digit. While frameworks like InfoGAN have addressed this by maximizing mutual information, they often rely on auxiliary "Q-networks" that can be difficult to stabilize.

This paper introduces Info-Sinkhorn, a novel GAN framework that achieves disentanglement by applying structured regularization directly to the Optimal Transport (OT) plan. By leveraging the theoretical rigor of OT and the computational efficiency of the Sinkhorn algorithm, the authors create a model where specific latent dimensions are forced to become highly informative without the instability of traditional auxiliary networks.

Methodology

The Info-Sinkhorn framework is built upon three primary innovations:

1. Debiased Sinkhorn Loss

Standard entropic OT losses often suffer from "variance collapse," where the generator produces blurry averages (means or medians) of data clusters rather than distinct samples. To combat this, the authors utilize a debiased version of the Sinkhorn loss. This ensures that the generator produces sharper, more diverse samples by preventing it from collapsing onto the mean of the data distribution.

Comparison of generated MNIST samples across different Optimal Transport formulations. Note the increased sharpness in the Sinkhorn (S) variants compared to standard entropic Wasserstein (W) variants.

2. Structured Entropic Regularization

The core innovation lies in manipulating the "transportation plan"—the map showing how latent codes connect to data points. The authors add a regularization term that minimizes the entropy of this plan for a designated subset of latent variables. Mathematically, this structured regularization is equivalent to maximizing the mutual information between those latent variables and the generated output.

3. Dual Interpretation as Ground Cost Learning

Through convex duality, the authors demonstrate that their regularization can be viewed as learning a ground cost. The model effectively learns to make "transporting" information through the designated "informative" latent dimensions cheaper than using the "noisy" ones, naturally guiding the model toward interpretable representations.

Key Results and Insights

MNIST Disentanglement

Evaluated on the MNIST dataset, Info-Sinkhorn successfully separated digit identity from style. Latent space traversals revealed that changing categorical codes cleanly swapped digit identity (e.g., '7' to '9') while maintaining style features like rotation and thickness. Conversely, adjusting continuous informative codes modified the slant or width without altering the digit's identity.

Transport Plan Sparsity

Qualitative analysis of the transportation plan confirms the effectiveness of the structured regularization. Without it (setting the regularization weight $\lambda=0$), the latent space remains diffuse and uninterpretable. With the proposed regularization (e.g., $\lambda=5$), the support of the transportation plan becomes sparse and structured, indicating that the model has successfully assigned specific meanings to specific latent dimensions.

Visualization of the transport plan $\pi$. The transition from (a) $\lambda=0$ to (b) $\lambda=5$ shows how structured regularization induces sparsity and interpretability in the latent-to-data mapping.

Conclusion

Info-Sinkhorn provides a theoretically grounded and practically efficient method for disentangled representation learning. By integrating mutual information maximization directly into the Optimal Transport objective via structured entropic regularization, the authors resolve common issues like variance collapse and training instability. While currently demonstrated on benchmarks like MNIST, future work aims to scale these OT-based informative constraints to high-resolution, complex image datasets.

Summary updated

Abstract

We tackle the challenge of disentangled representation learning in generative adversarial networks (GANs) from the perspective of regularized optimal transport (OT). Specifically, a smoothed OT loss gives rise to an implicit transportation plan between the latent space and the data space. Based on this theoretical observation, we exploit a structured regularization on the transportation plan to encourage a prescribed latent subspace to be informative. This yields the formulation of a novel informative OT-based GAN. By convex duality, we obtain the equivalent view that this leads to perturbed ground costs favoring sparsity in the informative latent dimensions. Practically, we devise a stable training algorithm for the proposed informative GAN. Our experiments support the hypothesis that such regularizations effectively yield the discovery of disentangled and interpretable latent representations. Our work showcases potential power of a regularized OT framework in the context of generative modeling through its access to the transport plan. Further challenges are addressed in this line.
BibTeX
@misc{brechet2019informative,
  author = {Pierre Bréchet and Tao Wu and Thomas Möllenhoff and Daniel Cremers},
  title = {{Informative GANs via Structured Regularization of Optimal Transport}},
  year = {2019},
  eprint = {1912.02160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.02160},
}
Figure