Comparison of the effects of attention mechanism on translation tasks of different lengths of ambiguous words

Yue Hu, Jiahao Qin, Zemeiqi Chen, Jingshi Zhou, Xiaojun Zhang2020 · DOI

Abstract

In recent years, the attention mechanism has become a cornerstone of neural machine translation (NMT) tasks based on the encoder-decoder architecture. This paper investigates the performance of this mechanism in word sense disambiguation (WSD) across varying text lengths, specifically examining the influence of context markers. Contrary to the initial hypothesis that attention mechanisms perform consistently across sentence lengths, our findings indicate that while the alignment effect is magnified in short-text translations involving ambiguous nouns, it diminishes significantly in long-text tasks. This suggests that the attention mechanism prioritizes ambiguous nouns over necessary context markers in longer sequences, leading to a performance decline as text length increases.

Introduction: The Challenge of Ambiguity

Ambiguity remains a primary hurdle in Natural Language Processing (NLP). Humans naturally utilize surrounding context to distinguish between different meanings of a word—for instance, determining whether "bank" refers to a financial institution or a riverbank. In NMT, the attention mechanism is theoretically designed to handle this by allowing the model to "attend" to relevant context markers throughout a sentence.

However, the scalability of this mechanism remains under-explored. This study addresses whether a model can disambiguate a word as effectively in an 80-word sentence as it does in a 10-word sentence. While researchers initially hypothesized consistent performance, experimental evidence reveals a significant limitation in how current architectures handle long-form context.

Methodology and Metrics

The study evaluated two distinct NMT architectures: Google T5 (a transformer-based model) and Martin MT (utilizing standard attention mechanisms). To quantify the effectiveness of attention across scales, the researchers introduced a residual coefficient ($R$). This metric measures the difference between a perfect diagonal alignment matrix (word-to-word mapping) and the actual attention weight matrix produced by the model.

Figure 1: The trend of the attention weight residual ($R$) relative to sentence length, showing increased dilution of focus in longer sequences.

As shown in Figure 1, a higher $R$ value indicates that attention is less concentrated on specific alignments. This suggests that the model's focus becomes increasingly "diluted" as the input sequence grows, making it harder to pinpoint specific context markers.

Experimental Setup

The evaluation utilized two primary datasets:

  • ContraWSD: A specialized dataset containing 5,000 test sentences featuring ambiguous nouns.
  • OPUS Parallel Corpus: A control group representing standard machine translation tasks without a specific focus on ambiguity.

Sentences were categorized into Short-Text (under 20 words) and Long-Text (40 to 80 words) to isolate the impact of length on disambiguation accuracy.

Results and Analysis

The experimental results demonstrated a stark contrast between standard translation and WSD tasks as sentence length increased. While translation quality for the control group (OPUS) remained relatively stable, the performance for ambiguous text (ContraWSD) experienced a precipitous decline.

Figure 2: BLEU score performance of NMT models across different datasets and sentence lengths, highlighting the sharp decline in WSD accuracy compared to the control.

Quantitative analysis using BLEU scores confirms this trend. For the T5 model, translation quality for ambiguous words dropped by nearly 70% in long-text scenarios, whereas the control group only saw a ~12% decrease.

Model-Dataset Short-Text BLEU Long-Text BLEU Mean BLEU
T5-ContraWSD 26.078 7.877 16.978
T5-OPUS 25.992 22.817 24.404
Martin-ContraWSD 25.323 12.479 18.901
Martin-OPUS 26.805 25.067 25.936

The Alignment vs. Context Conflict

The data suggests that in short sentences, the attention mechanism's "alignment effect" is highly effective at mapping ambiguous nouns to their correct translations. However, in longer sentences, the mechanism fails to "link" these nouns to distant context markers. Instead, the model appears to over-focus on the ambiguous noun itself, losing the signal required for accurate disambiguation amidst the increased "noise" of a longer sentence.

Conclusion

This study concludes that the attention mechanism is not the primary driver for integrating context information in long-form WSD tasks within NMT models. The observed "attention dilution" suggests that as text length increases, the mechanism struggles to prioritize relevant context markers over the ambiguous nouns themselves. Future research will focus on optimizing attention weights specifically for WSD in long-form text, exploring architectures that can better maintain the signal of context markers across extended sequences.

Summary updated

Abstract

In recent years, attention mechanism has been widely used in various neural machine translation tasks based on encoder decoder.This paper focuses on the performance of encoder decoder attention mechanism in word sense disambiguation task with different text length, trying to find out the influence of context marker on attention mechanism in word sense disambiguation task.We hypothesize that attention mechanisms have similar performance when translating texts of different lengths.Our conclusion is that the alignment effect of attention mechanism is magnified in short text translation tasks with ambiguous nouns, while the effect of attention mechanism is far less than expected in long-text tasks, which means that attention mechanism is not the main mechanism for NMT model to feed WSD to integrate context information.This may mean that attention mechanism pays more attention to ambiguous nouns than context markers.The experimental results show that with the increase of text length, the performance of NMT model using attention mechanism will gradually decline.
BibTeX
@misc{hu2020comparison,
  author = {Yue Hu and Jiahao Qin and Zemeiqi Chen and Jingshi Zhou and Xiaojun Zhang},
  title = {{Comparison of the effects of attention mechanism on translation tasks of different lengths of ambiguous words}},
  year = {2020},
  doi = {10.18653/v1/2020.iwdp-1.4},
}
Figure