Abstract
Training Variational Autoencoders (VAEs) for text is notoriously difficult because the model often learns to ignore the latent variables entirely—a phenomenon known as "KL vanishing." This paper introduces a cyclical annealing schedule for the KL penalty weight, repeatedly ramping it from zero to one. This simple tweak acts as a "warm restart," forcing the decoder to periodically re-engage with latent codes and resulting in significantly more informative representations for language modeling, dialog, and pre-training.
The KL Vanishing Problem
In a standard VAE, the goal is to compress data into a latent code ($z$) and then reconstruct the original data from that code. For text, we typically use an autoregressive decoder (like an LSTM), which generates words sequentially.
The authors identify that an autoregressive decoder has two competing ways to predict the next word:
- Path A: Utilizing the latent code $z$ (representing global semantic meaning).
- Path B: Utilizing the words it just generated (representing local context).
Because Path B is much easier to learn early in training, the model often "cheats" by relying solely on local context and ignoring the latent code. When this happens, the KL divergence term in the loss function drops to zero, the latent space becomes useless noise, and the model loses the ability to perform controllable generation. This is the "KL vanishing" problem.
The Cyclical Annealing Schedule
The standard mitigation strategy is monotonic annealing: starting the weight of the KL term ($\beta$) at 0 and slowly increasing it to 1. This gives the model a "head start" to learn Path A before the KL penalty kicks in. However, the authors argue that once $\beta$ reaches 1, the model can still get stuck in a state where Path A is blocked.
The authors propose the Cyclical Annealing Schedule. Instead of one long ramp-up, training is split into $M$ cycles. In each cycle:
- Annealing Phase: $\beta$ increases from 0 to 1, re-opening Path A and encouraging the model to find useful latent codes.
- Fixing Phase: $\beta$ stays at 1, forcing the model to optimize the true VAE objective and match the prior distribution.
By resetting $\beta$ to 0 multiple times, the model receives multiple "warm restarts." It leverages the representations learned in previous cycles to start the next one, progressively refining the latent space.
Experimental Evaluation
The authors evaluated their method on three distinct NLP tasks: Language Modeling (Penn Tree Bank), Dialog Response Generation (Switchboard), and Unsupervised Language Pre-training (Yelp).
Key Results
- Improved Latent Information: On PTB, cyclical annealing achieved higher KL values and better Perplexity (PPL) than monotonic annealing and other state-of-the-art VAE variants.
- Diverse Generation: In dialog tasks, the cyclical schedule produced significantly more diverse and relevant responses. While monotonic schedules often defaulted to generic phrases like "I'm not sure," the cyclical version generated specific, topical content.
- Better Downstream Features: Features learned via cyclical annealing on the Yelp dataset led to higher classification accuracy compared to standard VAEs or basic Autoencoders.
- Zero Computational Overhead: Cyclical annealing is a simple scheduling change that requires no extra compute power or architectural modifications.
Conclusion
Cyclical annealing provides a simple yet powerful solution to the KL vanishing problem. By treating the KL weight schedule as a series of warm restarts, it prevents the decoder from ignoring latent variables, leading to richer and more useful representations for a variety of natural language processing tasks.
Summary updated