Overview
Researchers from UMass Amherst and IIT Bombay have introduced Posterior Attention Models (PAM), a mathematically principled approach to how neural networks "pay attention" during sequence-to-sequence tasks like machine translation. By using the actual predicted output to refine the attention map (a posterior update) and treating the output as a mixture of probabilities rather than a mixture of inputs, PAM achieves significantly better alignment and translation quality than standard soft attention mechanisms.
The Problem with Standard Soft Attention
In standard "soft attention" models—the backbone of many early Transformers and RNN-based translators—the model calculates a weighted average of all input words to create a single "context vector." This vector is then used to predict the next output word.
The authors argue this approach is statistically unprincipled for two main reasons:
- Input Diffusion: Mixing all input words into a single averaged vector before predicting the output is akin to identifying an object in a blurry, averaged image. It loses the "hardness" or specificity of focusing on one word at a time.
- Lack of Feedback: Standard models calculate attention based only on the previous state. They fail to use the word they just predicted to correct or refine where they should be looking next.
The PAM Architecture
To resolve these issues, the paper proposes a "Posterior Attention Model" that introduces two fundamental architectural changes:
- From Mixture of Inputs to Mixture of Outputs: Instead of averaging the inputs and making one prediction, PAM makes a separate prediction for every possible input position and then averages those probabilities. This allows the model to retain the benefits of "hard attention" (focusing on specific words) while remaining differentiable and easy to train via backpropagation.
- Posterior Update: After the model predicts an output token, it uses Bayes' Rule to update its attention distribution. This "posterior" distribution is much sharper and more accurate because it accounts for the word actually chosen. This corrected "gaze" is then passed to the next step of the sequence.
- Structural Biases: The authors also experimented with adding "Proximity" and "Monotonicity" biases, encouraging the model to look at words near the previous focus or to move steadily from left to right.
Key Insights and Results
The models were evaluated on Machine Translation (five language pairs including German-English and Japanese-English) and Morphological Inflection tasks.
- Sharper Focus and Better Coverage: The entropy of posterior attention is much lower than standard soft attention, meaning the model is much more "certain" about which input word corresponds to which output word. Furthermore, PAM shows superior "input coverage," ensuring almost every word in the source sentence is eventually attended to.
- Self-Correction: The posterior update allows the model to correct mistakes on the fly. If the "prior" attention was looking at the wrong word, seeing the actual output token helps the model realize its mistake and re-align for the next step.
- Alignment Accuracy: On the RWTH German-English dataset, PAM improved the Alignment Error Rate (AER) by more than 10% over the next best model, proving it is significantly better at pointing to the correct source word.
- Translation Quality: PAM consistently outperformed soft and sparse attention models across all five translation tasks, yielding higher BLEU scores.
- Efficiency: Despite the rigorous mathematical foundation, the running time overhead is only about 40% over standard soft attention—a manageable trade-off for the notable gains in accuracy.
Limitations and Future Directions
While highly effective, PAM does face computational challenges. Calculating a separate softmax for every input position is expensive. The authors mitigated this using a "Top-K" approximation (only looking at the 6 most likely words), but scaling to massive vocabularies remains a hurdle.
Additionally, the current paper focuses on RNN-based encoder-decoder models. While the principles of posterior updates are universal, further research is needed to integrate these ideas with modern, multi-headed Transformer architectures or to incorporate more complex structural biases for multi-modal tasks.
Summary updated