Abstract
Sequence-to-sequence (seq2seq) models traditionally rely on deterministic soft attention mechanisms to generate context vectors. However, these deterministic representations often limit generalization and are susceptible to local optima. This paper presents Amortized Context Vector Inference (ACVI), a novel approach that treats context vectors as latent random variables modeled by a Gaussian mixture. By employing amortized variational inference, ACVI enhances the model's ability to generate novel content and improves performance across tasks such as abstractive text summarization and video captioning with minimal computational overhead.
Introduction
Sequence-to-sequence (seq2seq) architectures have become the standard for tasks involving variable-length input and output sequences, such as machine translation, document summarization, and video captioning. Central to these models is the Neural Attention (NA) mechanism, specifically Soft Attention (SA), which allows the decoder to focus on specific parts of the input sequence at each time step.
In standard SA, the context vector is computed as a deterministic weighted average of the encoder's hidden states. While effective, this deterministic approach has inherent limitations: it can lead the model to converge on poor local optima and often results in poor generalization when the model encounters unseen data or shifts in domain. Drawing inspiration from Bayesian deep learning, this paper proposes that introducing stochasticity into the attention mechanism can serve as a powerful regularizer, allowing for more robust and creative sequence generation.
Methodology: Amortized Context Vector Inference (ACVI)
The core contribution of this work is Amortized Context Vector Inference (ACVI). Rather than treating the context vector as a fixed point in space, ACVI redefines it as a latent random variable drawn from a finite mixture model.
The Gaussian Mixture Model
In the ACVI framework, the context vector is sampled from a Gaussian mixture model (GMM). The components of this mixture are Gaussian distributions centered around the encoder's hidden states. The variances for these Gaussians are not fixed; they are learned dynamically via a simple ReLU-activated layer.
A key mathematical insight of this paper is that standard Soft Attention is actually a special, "collapsed" case of ACVI. When the variance of the Gaussian components is set to zero, the ACVI sampling process reverts to the deterministic weighted average used in SA.
Variational Inference and Training
To train the model, the authors utilize amortized variational inference, maximizing the Evidence Lower Bound (ELBO). To ensure that the model remains end-to-end differentiable despite the sampling process, the standard reparameterization trick is applied to the Gaussian components.
One challenge in this approach is the computational cost associated with sampling from categorical distributions (the mixture weights), which typically requires the Gumbel-Softmax relaxation. To maintain the efficiency of standard attention, the authors implement a two-phase training heuristic:
- Phase 1 (First 90%): The model uses a simple weighted average heuristic.
- Phase 2 (Final 10%): The model switches to the full Gumbel-Softmax relaxation to fine-tune the stochastic behavior.
Experimental Setup
The authors evaluated ACVI across three distinct tasks and datasets:
- Abstractive Document Summarization (ADS): Evaluated on the non-anonymized CNN/Daily Mail dataset.
- Zero-Shot Domain Adaptation: Models trained on CNN/Daily Mail were tested on the DUC2004 dataset (500 English documents) without further training to test robustness.
- Video Captioning (VC): Evaluated on the Youtube2Text corpus, which contains approximately 80,000 video-description pairs.
Results and Discussion
Performance Gains
ACVI demonstrated superior performance across all primary benchmarks. In abstractive summarization on the CNN/Daily Mail dataset, ACVI achieved a ROUGE-1 score of 42.71, significantly outperforming standard Pointer-Generator networks (39.53) and other variational models like VED (41.28). Similar improvements were seen in video captioning, where ACVI reached a top CIDEr score of 0.4375.
Creativity and Novelty
One of the most striking findings was the impact of stochasticity on "novelty." By sampling from a distribution, the model is less likely to simply copy-paste segments of the source text. ACVI increased the rate of generated novel words by more than 3x compared to the next best variational method (a rate of 0.38 for ACVI vs. 0.12 for VED and a mere 0.05 for standard SA).
Robustness and Generalization
The variational regularization inherent in ACVI proved highly effective for domain adaptation. When tested on the DUC2004 dataset—a domain different from its training data—the ACVI model maintained a ROUGE-1 score of 32.09. In contrast, the baseline SA model dropped to 27.02, suggesting that ACVI helps the model learn more generalizable representations rather than overfitting to training-specific stylistic patterns.
Conclusion
Amortized Context Vector Inference provides an elegant, mathematically grounded extension to the standard attention mechanism. By treating context vectors as latent variables, ACVI introduces a beneficial form of Bayesian regularization that improves both the accuracy and the creativity of seq2seq models. While the current training regime requires a two-phase approach for maximum efficiency, the negligible overhead at inference time makes it a practical upgrade for real-world applications. Future research will focus on adapting ACVI to handle power-law (Zipfian) distributions, which could further refine the model's performance on the long tail of natural language vocabulary.
Summary updated