Dirichlet Variational Autoencoder for Text Modeling

Yijun Xiao, Tiancheng Zhao, William Yang WangarXiv 2018 · OpenReview

TL;DR

This paper introduces a novel Variational Autoencoder (VAE) for text modeling that explicitly incorporates topic distributions using a Dirichlet latent variable. This approach not only improves text reconstruction and representation learning but also effectively mitigates the common "KL divergence vanishing" problem in text VAEs, leading to more robust and informative latent representations.

What Problem Does This Paper Address?

Variational Autoencoders (VAEs) are powerful generative models for learning unsupervised representations of data. However, when applied to text, VAEs often suffer from a phenomenon called "KL divergence vanishing." This means the KL divergence term in the VAE's objective function collapses to zero during training, rendering the encoder useless and the decoder essentially just a language model. This effectively prevents the VAE from learning meaningful latent representations of the text. Existing solutions often involve complex architectural changes or modifications to the training objective. This paper aims to address this issue by introducing a more principled way to incorporate topic information into the VAE framework, making it more robust to KL divergence vanishing and improving its text modeling capabilities.

Methods

The core innovation of this paper is the introduction of a Dirichlet latent variable alongside the traditional multivariate Gaussian latent variable in a VAE. This Dirichlet variable is designed to explicitly model the topic distribution of a document. The generative process is envisioned as first determining an abstract meaning (Gaussian $z$), then using this to inform a topic distribution (Dirichlet $t$), and finally generating the text ($x$) based on both $z$ and $t$.

The authors propose two variants:

  • Joint Model: Treats the topic distribution $t$ as an observed variable, requiring it to be provided during training.
  • Marginal Model: Treats $t$ as an unobserved variable, allowing the model to infer topic distributions from the input text. This is the more practical and preferred approach as it doesn't require external topic modeling components.

The model's objective is to maximize a variational lower bound (ELBO) that accounts for both the reconstruction of the text and the KL divergence terms for both the Gaussian and Dirichlet latent variables. The authors derive the ELBO for both the joint and marginal models. They also specify the neural network architectures, using LSTMs for encoding and decoding, and MLPs to parameterize the distributions.

Datasets

The paper evaluates their proposed model on four datasets:

  • Penn Treebank (PTB): A dataset of relatively short sentences.
  • Book Corpus (BC): Another dataset with relatively short sentences.
  • Yahoo Answer (YAHOO): A dataset of longer documents with ten class labels.
  • Yelp15 review (YELP): A dataset of longer documents with five rating levels.

These datasets were chosen to assess the model's performance on both sentence-level and document-level text reconstruction and representation learning.

Key Insights and Ideas

  • Topic-Awareness Improves Reconstruction: Explicitly modeling topic distributions with a Dirichlet variable significantly enhances the VAE's ability to reconstruct input texts. This suggests that topics are a crucial component for understanding and generating coherent text.
  • Mitigating KL Divergence Vanishing: The interaction between the Dirichlet topic variable and the Gaussian latent variable naturally helps to prevent KL divergence vanishing. The need to predict topics accurately from the latent representation forces the encoder to retain meaningful information, thus keeping the KL divergence non-trivial.
  • Marginal Model's Practicality: The marginal model, which infers topic distributions, is a significant advantage as it doesn't require pre-trained topic models, making it a more self-contained and practical solution.
  • Latent Space Interpolation Reveals Control: Interpolating in the latent space (both $z$ and $t$) demonstrates that the Dirichlet variable $t$ primarily influences word choice and topic, while the Gaussian variable $z$ seems to control sentence structure and abstract meaning.
  • Optimal KL Divergence Exists: Experiments on a synthetic dataset suggest that there's an optimal level of KL divergence for VAEs. Too little KL leads to random representations, while too much leads to memorization. The proposed Dirichlet VAE naturally finds a better balance.

Key Results

  • Superior Reconstruction: The proposed Dirichlet VAE consistently outperforms standard VAEs and VAEs with bag-of-words (BOW) loss in terms of combined NLL+KL and perplexity (PPL) across all tested datasets. For example, on the PTB dataset, the proposed model achieves the lowest combined NLL+KL (2.7) and PPL (33.4).
  • Better Classification Performance: When used for classification tasks on learned representations from YAHOO and YELP datasets, the topic-aware VAE models achieve higher test accuracies compared to standard VAEs. This indicates that the learned latent representations are more discriminative.
  • Improved Conditional Generation: In CVAE settings, the proposed model maintains good performance even when generating low-probability samples, unlike models trained with BOW loss which degrade sharply.
  • Non-Trivial KL Divergence: Unlike standard VAEs on longer documents (YAHOO, YELP) which exhibit very small KL divergences (0.4-0.5), the proposed model consistently learns latents with non-trivial KL divergences across all datasets.

Limitations and Future Work

  • Optimal KL Divergence: The paper acknowledges that determining the optimal level of KL divergence for a given task and model setting remains an open problem.
  • Disentangled Representations: The authors suggest future work could focus on learning disentangled representations for $z$ and $t$, allowing for more precise control over generation.
  • Systematic Determination of KL Level: Future research could explore systematic methods for determining the proper KL divergence level based on model settings and applications.

Summary updated

Abstract

We introduce an improved variational autoencoder (VAE) for text modeling with topic information explicitly modeled as a Dirichlet latent variable. By providing the proposed model topic awareness, it is more superior at reconstructing input texts. Furthermore, due to the inherent interactions between the newly introduced Dirichlet variable and the conventional multivariate Gaussian variable, the model is less prone to KL divergence vanishing. We derive the variational lower bound for the new model and conduct experiments on four different data sets. The results show that the proposed model is superior at text reconstruction across the latent space and classifications on learned representations have higher test accuracies.
BibTeX
@misc{xiao2018dirichlet,
  author = {Yijun Xiao and Tiancheng Zhao and William Yang Wang},
  title = {{Dirichlet Variational Autoencoder for Text Modeling}},
  year = {2018},
  eprint = {1811.00135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.00135},
}
Figure