Abstract
In recent years, the attention mechanism has become a cornerstone of neural machine translation (NMT) tasks based on the encoder-decoder architecture. This paper investigates the performance of this mechanism in word sense disambiguation (WSD) across varying text lengths, specifically examining the influence of context markers. Contrary to the initial hypothesis that attention mechanisms perform consistently across sentence lengths, our findings indicate that while the alignment effect is magnified in short-text translations involving ambiguous nouns, it diminishes significantly in long-text tasks. This suggests that the attention mechanism prioritizes ambiguous nouns over necessary context markers in longer sequences, leading to a performance decline as text length increases.
Introduction: The Challenge of Ambiguity
Ambiguity remains a primary hurdle in Natural Language Processing (NLP). Humans naturally utilize surrounding context to distinguish between different meanings of a word—for instance, determining whether "bank" refers to a financial institution or a riverbank. In NMT, the attention mechanism is theoretically designed to handle this by allowing the model to "attend" to relevant context markers throughout a sentence.
However, the scalability of this mechanism remains under-explored. This study addresses whether a model can disambiguate a word as effectively in an 80-word sentence as it does in a 10-word sentence. While researchers initially hypothesized consistent performance, experimental evidence reveals a significant limitation in how current architectures handle long-form context.
Methodology and Metrics
The study evaluated two distinct NMT architectures: Google T5 (a transformer-based model) and Martin MT (utilizing standard attention mechanisms). To quantify the effectiveness of attention across scales, the researchers introduced a residual coefficient ($R$). This metric measures the difference between a perfect diagonal alignment matrix (word-to-word mapping) and the actual attention weight matrix produced by the model.
As shown in Figure 1, a higher $R$ value indicates that attention is less concentrated on specific alignments. This suggests that the model's focus becomes increasingly "diluted" as the input sequence grows, making it harder to pinpoint specific context markers.
Experimental Setup
The evaluation utilized two primary datasets:
- ContraWSD: A specialized dataset containing 5,000 test sentences featuring ambiguous nouns.
- OPUS Parallel Corpus: A control group representing standard machine translation tasks without a specific focus on ambiguity.
Sentences were categorized into Short-Text (under 20 words) and Long-Text (40 to 80 words) to isolate the impact of length on disambiguation accuracy.
Results and Analysis
The experimental results demonstrated a stark contrast between standard translation and WSD tasks as sentence length increased. While translation quality for the control group (OPUS) remained relatively stable, the performance for ambiguous text (ContraWSD) experienced a precipitous decline.
Quantitative analysis using BLEU scores confirms this trend. For the T5 model, translation quality for ambiguous words dropped by nearly 70% in long-text scenarios, whereas the control group only saw a ~12% decrease.
| Model-Dataset | Short-Text BLEU | Long-Text BLEU | Mean BLEU |
|---|---|---|---|
| T5-ContraWSD | 26.078 | 7.877 | 16.978 |
| T5-OPUS | 25.992 | 22.817 | 24.404 |
| Martin-ContraWSD | 25.323 | 12.479 | 18.901 |
| Martin-OPUS | 26.805 | 25.067 | 25.936 |
The Alignment vs. Context Conflict
The data suggests that in short sentences, the attention mechanism's "alignment effect" is highly effective at mapping ambiguous nouns to their correct translations. However, in longer sentences, the mechanism fails to "link" these nouns to distant context markers. Instead, the model appears to over-focus on the ambiguous noun itself, losing the signal required for accurate disambiguation amidst the increased "noise" of a longer sentence.
Conclusion
This study concludes that the attention mechanism is not the primary driver for integrating context information in long-form WSD tasks within NMT models. The observed "attention dilution" suggests that as text length increases, the mechanism struggles to prioritize relevant context markers over the ambiguous nouns themselves. Future research will focus on optimizing attention weights specifically for WSD in long-form text, exploring architectures that can better maintain the signal of context markers across extended sequences.
Summary updated