Fixing Gaussian Mixture VAEs for Interpretable Text Generation

Wenxian Shi, Hao Zhou, Ning Miao, Shenjian Zhao, Lei LiarXiv 2019 · arXiv

Introduction

Variational Auto-encoders (VAEs) have become a cornerstone of natural language generation by mapping discrete text into a continuous latent space. While standard VAEs typically employ a single Gaussian prior, this approach lacks interpretability; distinct semantic categories, such as "greetings" versus "weather requests," are often lost in an undifferentiated cloud of points. Gaussian Mixture Variational Auto-encoders (GMVAEs) were proposed to solve this by using Gaussian Mixture Models (GMMs) as priors, where each mode represents a specific semantic category.

However, in practice, GMVAEs frequently suffer from "mode-collapse." During training, the standard Evidence Lower Bound (ELBO) objective mathematically incentivizes Gaussian components to overlap and merge. This results in the model ignoring the discrete mixture structure and degenerating back into a standard, uninterpretable VAE.

Methodology: The DGMVAE Framework

To address the inherent mathematical tendency toward collapse, this research introduces Dispersed-GMVAE (DGMVAE). The model utilizes a latent variable dependency structure where a discrete variable selects a Gaussian component, and a continuous variable determines the specific point within that component.

The core innovation of DGMVAE lies in two novel regularization terms added to the objective function:

  1. Mutual Information Term ($L_{mi}$): This term ensures that the discrete latent variable remains strongly correlated with the input data, preventing the model from "ignoring" the mixture components.
  2. Variance Regularization Term ($L_{var}$): This acts as a "dispersion force," explicitly penalizing components for clustering too closely together. This ensures the latent space maintains a multi-modal structure that is both interpretable and smooth enough for high-quality synthesis.

Experimental Results and Visual Proof

DGMVAE was evaluated across three benchmarks: Penn Treebank (PTB), Daily Dialogs (DD), and Stanford Multi-Domain Dialog (SMD). The results demonstrate that DGMVAE significantly outperforms standard GMVAEs in both language modeling performance and semantic interpretability.

On the PTB dataset, DGMVAE achieved a reverse Perplexity (rPPL) of 244.30, a massive improvement over the vanilla GMVAE's 923.66. Furthermore, homogeneity scores on the Daily Dialog dataset rose from 0.08 to 0.23, proving that the learned clusters align much more closely with human-labeled dialog acts.

As shown in the visualizations above, the standard GMVAE latent space is a disorganized mixture where different intents overlap. In contrast, DGMVAE successfully maintains distinct clusters, allowing for controlled and interpretable text generation where specific regions of the latent space correspond to clear semantic intents, such as scheduling meetings or asking about the weather.

Conclusion

DGMVAE provides a robust solution to the mode-collapse problem in mixture-based VAEs. By introducing dispersion-focused regularization, the model preserves a structured latent space without sacrificing the expressive capacity required for natural language. Future work will focus on scaling these algorithmic improvements to modern Transformer-based architectures and exploring automated methods for discovering the optimal number of semantic modes.

Summary updated

Abstract

Deep generative models are commonly used for generating images and text. Interpretability of these models is one important pursuit, other than the generation quality. Variational auto-encoder (VAE) with Gaussian distribution as prior has been successfully applied in text generation, but it is hard to interpret the meaning of the latent variable. To enhance the controllability and interpretability, one can replace the Gaussian prior with a mixture of Gaussian distributions (GM-VAE), whose mixture components could be related to hidden semantic aspects of data. In this paper, we generalize the practice and introduce DEM-VAE, a class of models for text generation using VAEs with a mixture distribution of exponential family. Unfortunately, a standard variational training algorithm fails due to the mode-collapse problem. We theoretically identify the root cause of the problem and propose an effective algorithm to train DEM-VAE. Our method penalizes the training with an extra dispersion term to induce a well-structured latent space. Experimental results show that our approach does obtain a meaningful space, and it outperforms strong baselines in text generation benchmarks. The code is available at https://github.com/wenxianxian/demvae.
BibTeX
@misc{shi2019fixing,
  author = {Wenxian Shi and Hao Zhou and Ning Miao and Shenjian Zhao and Lei Li},
  title = {{Fixing Gaussian Mixture VAEs for Interpretable Text Generation}},
  year = {2019},
  eprint = {1906.06719},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.06719},
}
Figure