Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation

Verna Dankers, Christopher G. Lucas, Ivan Titov

Abstract

Unlike literal expressions, idioms’ meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation - i.e. identifies the expression as idiomatic - the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms’ parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder’s cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer’s tendency to process idioms as compositional expressions contributes to literal translations of idioms.

Introduction

Neural Machine Translation (NMT) has reached impressive heights, but idioms remain a major stumbling block. Idioms are non-compositional: their meaning cannot be derived by simply adding up the meanings of their individual words. The dominant Transformer architecture, however, is built to be highly compositional, excelling at understanding how words relate to one another to form complex meanings.

This strength becomes a weakness when encountering expressions like "spill the beans." A model that is "too compositional" will translate each word literally, missing the figurative meaning. This research investigates whether the Transformer's internal mechanics—specifically its attention patterns and hidden representations—reflect this struggle between literal and figurative processing.

Methodology

The researchers analyzed pre-trained Transformer models (Marian-MT) translating from English into seven European languages (German, Swedish, Danish, French, Italian, Spanish, and Dutch). They utilized the MAGPIE Corpus, containing over 37,000 English idiom samples annotated as "fully figurative" or "fully literal."

To understand the model's internal state, the authors employed several interpretability techniques:

  • Heuristic Labeling: Categorizing outputs as "word-for-word" (literal/failed) or "paraphrase" (figurative/successful).
  • Attention Analysis: Mapping attention flow within the encoder (self-attention) and between the encoder and decoder (cross-attention).
  • Canonical Correlation Analysis (CCA): Comparing hidden representations of idioms across the six layers of the Transformer encoder.
  • Amnesic Probing (INLP): A causal intervention where "idiom-related" features were identified and "erased" to see if the model would revert to literal translations.

Internal Mechanics of Idiom Processing

Figure 1: Conceptual diagram of attention shifts when a Transformer identifies an idiom. (1) Reduced attention to context, (2) increased self-attention within the idiom, (3) reduced cross-attention to source tokens, and (4) increased attention to the end-of-sentence token.

The study reveals that when a model successfully translates an idiom figuratively, its internal mechanics undergo a distinct shift:

  1. The "Unitary" Shift: The encoder's self-attention begins treating the idiom's parts as a single lexical unit. Attention within the idiom increases, while interaction with the rest of the sentence decreases.
  2. Contextual Isolation: Paradoxically, the model "shuts out" the context when translating figuratively, processing the idiom in a more stand-alone manner.
  3. Decoder "Autopilot": In the cross-attention phase, the decoder pays less attention to specific source tokens and more to the end-of-sentence (</s>) token. This suggests the decoder is retrieving a stored paraphrase from its internal language model rather than translating the source words directly.

Empirical Results

Figure 6: Distribution of attention across layers for different translation outcomes: figurative paraphrases (fig-par), figurative (fig), literal (lit), and literal word-for-word (lit-wfw).
  • High Failure Rate: Even in state-of-the-art models, over 76% of figurative idioms were translated literally, highlighting a severe "compositionality trap."
  • Layer-wise Emergence: Probing showed that the "knowledge" of whether an expression is idiomatic increases as information moves to the higher layers of the encoder.
  • Causal Evidence: By using amnesic probing to remove "idiom features," the researchers successfully turned idiomatic translations back into literal ones in 27% to 40% of cases. This proves that the identified features are directly responsible for the model's ability to handle figurative language.
  • Cross-Lingual Consistency: These patterns were remarkably consistent across all seven target languages studied, regardless of their genetic similarity to English.

Conclusion

The study confirms that Transformers have a strong inductive bias toward compositionality. Even when the encoder correctly identifies a figurative use, the model often defaults to a literal translation because its architecture is so focused on word-by-word relationships. To improve NMT performance on figurative language, future work may need to move away from "pure" compositionality by designing specific attention heads for multi-word expressions or introducing training objectives that penalize overly literal translations of known idioms.

Summary updated

Abstract

Unlike literal expressions, idioms’ meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation - i.e. identifies the expression as idiomatic - the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms’ parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder’s cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer’s tendency to process idioms as compositional expressions contributes to literal translations of idioms.
BibTeX
@inproceedings{dankers2022can,
  author = {Verna Dankers and Christopher G. Lucas and Ivan Titov},
  title = {{Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=T2eSdFQxkO},
}
Figure