Abstract
Unlike literal expressions, idioms' meanings do not directly follow from their parts, posing a challenge for neural machine translation (NMT). NMT models are often unable to translate idioms accurately and over-generate compositional, literal translations. In this work, we investigate whether the non-compositionality of idioms is reflected in the mechanics of the dominant NMT model, Transformer, by analysing the hidden states and attention patterns for models with English as source language and one of seven European languages as target language. When Transformer emits a non-literal translation—i.e. identifies the expression as idiomatic—the encoder processes idioms more strongly as single lexical units compared to literal expressions. This manifests in idioms' parts being grouped through attention and in reduced interaction between idioms and their context. In the decoder's cross-attention, figurative inputs result in reduced attention on source-side tokens. These results suggest that Transformer's tendency to process idioms as compositional expressions contributes to literal translations of idioms.
Introduction
The Transformer architecture has revolutionized Neural Machine Translation (NMT) through its ability to model complex, compositional relationships between words. However, this strength becomes a liability when encountering non-compositional language: idioms. Expressions like "kick the bucket" or "piece of cake" cannot be understood by summing the meanings of their individual components.
This research investigates a fundamental tension: is the Transformer's failure to translate idioms a byproduct of its excellence at compositionality? By analyzing the internal mechanics of pre-trained Marian-MT models across seven European languages, the study reveals how the model's "default" compositional mode often overrides idiomatic understanding, leading to a persistent literal bias.
The Problem: The Literal Bias
NMT models frequently struggle with idiomaticity, defaulting to "word-for-word" translations. For instance, translating "kick the bucket" as a physical act of hitting a pail rather than the figurative meaning "to die." The researchers found that even when a model possesses the internal information to recognize an idiom, it defaults to a literal translation in over 76% of cases across the tested languages (Dutch, German, Swedish, Danish, French, Italian, and Spanish).
Methodology
To understand why this happens, the authors employed a multi-pronged interpretability approach:
- Attention Analysis: Measuring self-attention within the encoder and cross-attention in the decoder.
- Probing Classifiers: Training diagnostic models to determine if hidden states contain enough information to distinguish literal from figurative usage.
- Amnesic Probing (INLP): A causal intervention where figurative information is "erased" from hidden states to observe if the model reverts to literal outputs.
- Heuristic Labeling: Categorizing model outputs as "word-for-word" (literal) or "paraphrase" (figurative) to track performance.
The study utilized the MAGPIE corpus, which contains 1,482 English Potentially Idiomatic Expressions (PIEs) across 37,000 samples, and the OPUS parallel corpora.
Key Findings: The "Unit Strategy"
The research identifies a specific mechanical shift that occurs when a Transformer successfully translates an idiom figuratively.
1. Internal Grouping
In successful idiomatic translations, the encoder increases attention within the idiom’s tokens and decreases attention to the surrounding context. Essentially, the model attempts to "turn off" its compositional nature, treating the multi-word expression as a single lexical unit.
2. Decoder Independence
When processing figurative inputs, the decoder relies less on specific source-side tokens. Instead, it shows increased cross-attention to the </s> (end-of-sentence) token. This suggests that the decoder shifts from a word-by-word translation mode to a language modeling mode, retrieving a stored paraphrase from its own internal weights rather than mapping directly from source words.
3. Semantic Maturity in Higher Layers
Probing results indicate that the ability to distinguish between literal and figurative usage is significantly stronger in the higher layers of the encoder. This confirms that the model treats idiomaticity as a high-level semantic property, yet this understanding often fails to propagate into the final output due to the "gravity" of the compositional default.
Causal Evidence
Using amnesic probing, the researchers demonstrated a causal link between these internal representations and the model's output. By removing the features identified by the probes as "figurative," the model reverted to literal translations in 27% to 40% of cases. This proves that the specific hidden features the researchers identified are indeed what drive the model's rare successful idiomatic translations.
Conclusion and Future Directions
The study concludes that the Transformer is, in a sense, "too compositional." Its architecture is so finely tuned to build meaning from parts that it struggles to maintain the "unit" representation required for idioms.
Future improvements in NMT might involve:
- Architectural Adjustments: Developing specialized attention heads that can more easily group multi-word expressions.
- Training Objectives: Implementing loss functions that penalize overly compositional translations for known non-compositional phrases.
By understanding the mechanical struggle between compositionality and idiomaticity, researchers can better design models that handle the full nuance of human language.
Summary updated