When sequence-to-sequence models generate text, it is tempting to assume that their internal attention weights map directly to linguistic alignments. For instance, when translating a sentence, we intuitively expect the model to assign the highest attention weight to the exact source word it is currently translating.
However, a substantial body of research demonstrates a fundamental gap between the fuzzy internal weights of a neural network and the structured, explicit mappings required for precise token alignment. As Koehn et al. observed in Six Challenges for Neural Machine Translation, attention weights do not always correspond to linguistic word alignment and should not be treated as a direct proxy for a model's reasoning. Understanding why this discrepancy occurs—and how to fix it—is crucial for building interpretable and controllable sequence models.
Why Attention and Alignment Diverge
To understand why the attention-alignment gap exists, we must look at what attention is actually optimized to do. In modern architectures like the Transformer, attention is not a dedicated alignment module; rather, it is a mathematical byproduct of optimizing for next-token prediction (Zuchao Li et al.). Because of this, attention heads are forced to multitask, leading to several distinct phenomena that degrade alignment precision.
1. Signal Dispersion Across Deep Layers
In deep Transformer architectures, the alignment signal becomes distributed and diluted. As Xintong Li demonstrates in On the Word Alignment from Neural Machine Translation, different layers prioritize entirely different linguistic features. While some layers focus on local word-to-word mappings, others track broad contextual relationships or complex syntactic structures like subject-verb agreement. This functional diversity creates structural noise, meaning that the model's internal understanding of word mapping is scattered across the network rather than localized in a single attention head.
2. Punctuation and Stop-Word Sinks
Standard softmax attention mechanisms often produce diffuse or "blurry" distributions that focus heavily on semantically irrelevant tokens, such as punctuation, stop words, or end-of-sentence tags (Stacey et al.; Zhang et al.). When a model does not need specific source information to generate a common target token, the softmax function tends to dump its remaining probability mass onto these frequent baseline tokens. This creates noisy attention maps that obscure the actual decision-making logic.
3. Continuous Averaging vs. Discrete Mapping
Standard attention uses soft, continuous weights to compute a weighted average of input states (Shijie Wu et al.). While this allows for end-to-end gradient descent, it avoids making explicit, discrete decisions about which specific input symbol corresponds to an output symbol. This continuous smoothing makes it difficult for models to capture the discrete nature of token-to-token or word-to-region alignments (Vidal).
Soft Attention (Continuous & Blurry):
Target Word ---> [ Source Word A (0.3) | Source Word B (0.2) | Punctuation (0.5) ]
True Alignment (Discrete & Sharp):
Target Word ---> [ Source Word A (1.0) ]
The Practical Consequences of Misalignment
The attention-alignment discrepancy is not just a theoretical problem; it introduces severe failure modes in downstream applications:
- Low-Resource Vulnerability: In data-scarce environments, attention distributions become even more smeared and erratic, failing to lock onto specific relevant words (Goindani et al.; Araabi et al.). Furthermore, when training on partially parallel corpora, forcing attention onto unaligned segments introduces noise that degrades the model's ability to learn local translation mechanics (Wang et al.).
- Failure Under Constraints: Tasks like terminology injection or automatic post-editing require precise token placement. If attention weights are diffuse or inaccurate, the system can easily misplace, duplicate, or entirely omit required terms (Bérard et al.; Hasler et al.).
- The Interpretability Illusion: Because attention maps are noisy and frequently point to previous tokens or punctuation rather than the semantic source, treating attention as an explanation for model behavior can be deeply misleading (Stahlberg et al.; John Wu et al.).
Bridging the Gap
To resolve this discrepancy, researchers have developed techniques to either extract cleaner alignments from standard networks or force attention mechanisms to behave more like traditional statistical aligners.
Explicit Alignment Objectives
One prominent solution is to introduce training objectives that explicitly penalize blurry or asymmetric attention. For example, Cheng et al. introduced an agreement-based joint training framework that forces source-to-target and target-to-source attention mechanisms to agree on their word alignments, significantly sharpening the resulting attention matrices. Other approaches introduce dedicated, supervised alignment heads trained on small amounts of gold-standard human alignments to handle constrained tasks (Kai Song et al.).
Target Foresight
Another intuitive mechanism is "Target Foresight," proposed by Peter et al.. Standard attention must guess which source words are relevant based only on the past generated target context. Target Foresight gives the attention mechanism a "sneak peek" at the actual target word it is trying to generate during training. This extra information grounds the attention weights, resulting in massive improvements in Alignment Error Rate (AER).
Beyond Raw Weights: Norm-Based Analysis
Rather than changing how models are trained, some researchers have changed how we read their internal states. Kobayashi et al. argue that analyzing raw attention weights ignores the magnitude of the transformed vectors being passed through the network. By introducing norm-based analysis—which multiplies the attention weight by the norm of the input vector—they demonstrated that Transformers are actually much better at filtering out punctuation noise than raw weights suggest, uncovering highly accurate alignments hidden within standard models.
Similarly, Linxin Song et al. showed that by filtering out the noise from irrelevant attention heads and focusing post-hoc explanations only on heads that track core syntax, token attribution accuracy can be dramatically improved.
Key Foundational Work
Neural Machine Translation with Supervised Attention (2016) — contributed 2 concept elements to this direction.
Agreement-Based Joint Training for Bidirectional Attention-Based Neural Machine Translation (2016) — contributed 2 concept elements to this direction.
What does Attention in Neural Machine Translation Pay Attention to? (2017) — contributed 2 concept elements to this direction.