Abstract
The Attentive Unsupervised Text (W)riter (AUTR) is a generative model that treats sentence construction as an iterative process on a dynamic canvas. By employing an external "canvas" memory and a modified attention mechanism, AUTR addresses the "posterior collapse" issue prevalent in text Variational Autoencoders (VAEs). This architecture encourages meaningful latent representations, leading to improved sentence generation and missing-word imputation performance.
Introduction: The Challenge of Posterior Collapse
In Variational Autoencoders (VAEs) for natural language processing, models often suffer from posterior collapse. In standard Recurrent Neural Network (RNN) based VAEs, the decoder frequently ignores the latent vector ($z$) provided by the encoder, relying instead on its own strength as a language model. While researchers have tried "word dropout" to force the model to rely on $z$, this is a workaround rather than a structural fix. The AUTR model proposes a structural shift: changing the generation process itself to make the latent representation indispensable.
Methodology: The Attentive Unsupervised Text (W)riter
Inspired by the DRAW model for image generation, AUTR "paints" sentences onto a 2D memory array called a canvas. Unlike sequential models that generate text from left to right, AUTR utilizes an iterative refinement process:
- Canvas Memory: The model maintains a 2D memory array where each "slot" represents a position in the sentence.
- Iterative Refinement: Over $T$ time steps, an RNN processes the latent vector $z$ and the current canvas state to determine where to "write" next.
- Modified Attention: A specialized softmax mechanism tracks slot occupancy, encouraging the model to fill empty positions and preventing redundant overwriting.
- Global-to-Local Generation: The model establishes global sentence structure before finalizing local word choices, ensuring the latent vector $z$ remains the primary guide.
Visualizing the Generation Process
One of the most compelling aspects of AUTR is the ability to observe the model's "thought process." By visualizing the canvas at intermediate steps, we can see the model first determining sentence length, then identifying key terms, and finally refining the grammatical structure.
This non-sequential approach allows the model to generate sentences of length $L$ in fewer than $L$ steps ($T < L$), as it can update multiple slots simultaneously. This efficiency is a key advantage over traditional step-by-step RNNs.
Experimental Results
The model was evaluated on the Book Corpus dataset, consisting of over 11,000 unpublished books. AUTR demonstrated superior performance compared to standard Gen-RNN baselines across several metrics.
As shown in the results, AUTR maintains a significantly higher KL divergence without requiring dropout, indicating that the latent space is more effectively utilized. It achieved a lower perplexity (PPL) of 37.4 compared to 41.9 for the baseline. Furthermore, in missing-word imputation tasks, AUTR recovered 34.1% of missing words, outperforming the baseline's 31.9%.
Conclusion and Future Work
AUTR successfully breaks the sequential constraints of traditional text generation, forcing the model to rely on global context provided by the latent space. While currently limited by a fixed canvas size and simple Markovian local dependencies, the model's ability to capture deep semantic structure suggests potential for broader applications in translation and question answering. Future iterations may explore dynamic canvas sizing and more sophisticated local dependency layers.
Summary updated