Standard attention mechanisms are remarkably effective at finding relevant information in a source sequence, but they suffer from a fundamental "memory" problem: they are often unaware of their own history. At each decoding step, the model looks at the entire input anew, with no built-in mechanism to remember which parts it has already translated or processed.
This lack of accounting leads to two classic failure modes in sequence-to-sequence models: over-translation (repeating the same word or phrase) and under-translation (skipping parts of the input entirely). To solve this, researchers have developed various "coverage" and "history-tracking" mechanisms that allow models to maintain a stateful record of their progress.
The Problem of Memoryless Attention
In a vanilla encoder-decoder model, the attention weights at time $t$ are calculated based on the current decoder state and the encoder hidden states. Crucially, the weights at time $t$ do not explicitly depend on the weights at time $t-1$.
As identified by Feng Shi, Shujie Liu, and colleagues in Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model, this causes the model to "forget" its alignment history. Without a sense of which words have already been "consumed," the model may get stuck in a loop or lose its place in a long sentence. This is particularly problematic for tasks requiring global coherence, such as generating recipes or long-form dialogue, where Chloé Kiddon and Yejin Choi note that models struggle to follow a specific "agenda" without an explicit accounting state (Globally Coherent Text Generation with Neural Checklist Models).
Tracking Progress with Coverage Vectors
The most direct solution to this problem is the coverage mechanism. Introduced by Zhaopeng Tu, Yang Liu, and others in Modeling Coverage for Neural Machine Translation, this approach maintains a "coverage vector" that accumulates the attention weights assigned to each source word over time.
Think of this as a checklist:
- Accumulation: Every time a source word receives attention, its corresponding value in the coverage vector increases.
- Feedback: This vector is fed back into the attention scoring function for the next step.
- Penalty: If a word has already received high cumulative attention (meaning it has likely been translated), the model is penalized for attending to it again.
This mechanism ensures that every part of the input is processed exactly once. This intuition extends beyond text; for example, Shuiling Zeng and Zhaoxian Li applied coverage to chemical structure recognition, ensuring that every atom and bond in an image is accounted for in the generated SMILES string (Chemical structure recognition method based on attention mechanism and encoder-decoder architecture).
Input Feeding and Recurrent Attention
A more architectural approach to history tracking is input-feeding. In Effective Approaches to Attention-based Neural Machine Translation, Thang Luong and Christopher D Manning proposed concatenating the attentional context vector from the previous step with the input of the current step.
By "feeding" the previous alignment decision back into the model, the decoder gains a memory of what it just did. This simple feedback loop helps the system manage complex word re-ordering and has become a staple in speech recognition as well, as demonstrated by Chao Weng and Dong Yu (Improving Attention Based Sequence-to-Sequence Models for End-to-End English Conversational Speech Recognition).
Modeling the Past and the Future
Some researchers have taken this a step further by explicitly partitioning the model's understanding of time. In Modeling Past and Future for Neural Machine Translation, Hao Zhou and Lili Mou introduced dedicated layers to track the PAST (content already translated) and the FUTURE (content yet to be addressed).
This dual-stream approach provides the generator with a continuous global view of its progress. Similarly, Junyang Lin and Xu Sun developed Adaptive Control of Attention (ACA), which uses a recurrent memory to dynamically gate the attention signal based on the decoding history (Decoding-History-Based Adaptive Control of Attention for Neural Machine Translation).
Why It Works: Fertility and Focus
The success of these methods stems from their ability to mimic human-like translation constraints. In traditional statistical machine translation, "fertility" refers to how many target words a single source word should produce. Coverage modeling provides a neural equivalent to this concept.
By maintaining a stateful record—whether through a checklist (Kiddon et al.), a coverage vector (Tu et al.), or a history-tracking vector (Zhang et al.)—the model transforms the attention mechanism from a simple "search" tool into a "planning" tool. This ensures that the decoder systematically processes the intended information, leading to outputs that are both more complete and less redundant.