Better Exploiting Latent Variables in Text Modeling

Canasai KruengkraiACL 2019 · DOI

Introduction

Variational Autoencoders (VAEs) are a powerful class of generative models that map high-dimensional data, such as text, into a continuous latent space. In natural language processing, VAEs are typically paired with recurrent neural network (RNN) decoders, like LSTMs, to capture the sequential nature of language. However, these models frequently encounter "posterior collapse," where the decoder learns to ignore the latent variable $z$ entirely, relying solely on its own autoregressive properties. This results in a KL-divergence near zero and a latent space that fails to encode meaningful semantic information.

Proposed Method: LSTM-VAE-AVG

To address posterior collapse and better utilize latent information, this work introduces a simple modification to the standard VAE architecture. The core innovation is Hidden State Averaging (LSTM-VAE-AVG).

Instead of drawing a single latent sample $z$, the model draws $L$ independent samples from the approximate posterior $q(z|x)$. Rather than simply averaging the final loss, the model maintains $L$ parallel decoding paths. At each time step $t$, the hidden states from all $L$ paths are averaged to form a single representative state $\tilde{h}_t$, which is then used to predict the next token.

The LSTM-VAE-AVG architecture averages hidden states across multiple latent samples at each decoding step.

By forcing the decoder to operate on an averaged hidden state, the model is incentivized to find a region in the latent space that consistently supports the target sequence, making the latent variables more robust and influential during the generation process.

Experimental Results

The proposed method was evaluated on the Penn Treebank (PTB) and Yahoo datasets. The results demonstrate that LSTM-VAE-AVG significantly outperforms baseline VAEs and other specialized techniques designed to prevent posterior collapse.

Quantitative Performance

As shown in the table below, the LSTM-VAE-AVG model achieves superior perplexity (PPL) and maintains a healthy KL divergence, indicating that the latent space is being actively utilized.

Comparison of model performance on the Yahoo dataset. The AVG model shows significant improvements in PPL and KL divergence.

On the Yahoo dataset, the model reached a PPL of 60.3 with $L=5$, surpassing more complex methods like the Semi-Amortized VAE (SA-VAE). Notably, the KL divergence for the AVG model is significantly higher (around 10-12) compared to the baseline VAE (1.5), confirming that the model successfully avoids posterior collapse.

Robustness in Inputless Settings

The effectiveness of the latent variables is further highlighted in the "inputless" setting, where the decoder does not receive the ground-truth previous word as input. In this challenging scenario, the LSTM-VAE-AVG model maintains much lower perplexity than the baseline, proving that the latent vector $z$ provides critical semantic guidance that the decoder can no longer ignore.

Conclusion

The LSTM-VAE-AVG approach offers a straightforward yet highly effective solution to the problem of posterior collapse in text modeling. By integrating multiple latent samples directly into the decoder's recurrent state through averaging, the model achieves state-of-the-art performance while ensuring the latent space remains semantically rich. This method serves as a robust baseline for future research in variational text generation.

Summary updated

BibTeX
@inproceedings{kruengkrai2019better,
  author = {Canasai Kruengkrai},
  title = {{Better Exploiting Latent Variables in Text Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1553},
}
Figure