Introduction
Variational Autoencoders (VAEs) are a powerful class of generative models that map high-dimensional data, such as text, into a continuous latent space. In natural language processing, VAEs are typically paired with recurrent neural network (RNN) decoders, like LSTMs, to capture the sequential nature of language. However, these models frequently encounter "posterior collapse," where the decoder learns to ignore the latent variable $z$ entirely, relying solely on its own autoregressive properties. This results in a KL-divergence near zero and a latent space that fails to encode meaningful semantic information.
Proposed Method: LSTM-VAE-AVG
To address posterior collapse and better utilize latent information, this work introduces a simple modification to the standard VAE architecture. The core innovation is Hidden State Averaging (LSTM-VAE-AVG).
Instead of drawing a single latent sample $z$, the model draws $L$ independent samples from the approximate posterior $q(z|x)$. Rather than simply averaging the final loss, the model maintains $L$ parallel decoding paths. At each time step $t$, the hidden states from all $L$ paths are averaged to form a single representative state $\tilde{h}_t$, which is then used to predict the next token.
By forcing the decoder to operate on an averaged hidden state, the model is incentivized to find a region in the latent space that consistently supports the target sequence, making the latent variables more robust and influential during the generation process.
Experimental Results
The proposed method was evaluated on the Penn Treebank (PTB) and Yahoo datasets. The results demonstrate that LSTM-VAE-AVG significantly outperforms baseline VAEs and other specialized techniques designed to prevent posterior collapse.
Quantitative Performance
As shown in the table below, the LSTM-VAE-AVG model achieves superior perplexity (PPL) and maintains a healthy KL divergence, indicating that the latent space is being actively utilized.
On the Yahoo dataset, the model reached a PPL of 60.3 with $L=5$, surpassing more complex methods like the Semi-Amortized VAE (SA-VAE). Notably, the KL divergence for the AVG model is significantly higher (around 10-12) compared to the baseline VAE (1.5), confirming that the model successfully avoids posterior collapse.
Robustness in Inputless Settings
The effectiveness of the latent variables is further highlighted in the "inputless" setting, where the decoder does not receive the ground-truth previous word as input. In this challenging scenario, the LSTM-VAE-AVG model maintains much lower perplexity than the baseline, proving that the latent vector $z$ provides critical semantic guidance that the decoder can no longer ignore.
Conclusion
The LSTM-VAE-AVG approach offers a straightforward yet highly effective solution to the problem of posterior collapse in text modeling. By integrating multiple latent samples directly into the decoder's recurrent state through averaging, the model achieves state-of-the-art performance while ensuring the latent space remains semantically rich. This method serves as a robust baseline for future research in variational text generation.
Summary updated