Food for Thought: How can we exploit contextual embeddings in the translation of idiomatic expressions?

Lukas Santing, Ryan Sijstermans, Giacomo Anerdi, Pedro Jeuris, Marijn ten Thij, Riza Batista-Navarro2022 · DOI

TL;DR

Researchers investigated how to improve the translation of idiomatic expressions—notoriously difficult for AI because their meanings aren't literal—using the Text-to-Text Transfer Transformer (T5) model. They found that while a pipeline approach (paraphrasing an idiom into plain English before translating) is effective, directly fine-tuning a T5 model on idiom-specific parallel data yields the most fluent and accurate English-to-German translations.

What Problem Does This Paper Address?

Idioms like "breaking the ice" or "in a nutshell" are the "final boss" of machine translation (MT). Their meaning is non-compositional, meaning you can't figure out the definition just by looking at the individual words.

Standard MT models often fall into the "literal trap." For example, a model might translate "Don't beat around the bush" into a German sentence about physically hitting shrubbery. While Transformer models and contextual embeddings have revolutionized NLP tasks like sentiment analysis, idiom translation has lagged behind. This is largely due to two factors:

  1. Ambiguity: An expression can be idiomatic in one context ("He broke the ice at the party") and literal in another ("He broke the ice for his soda").
  2. Data Scarcity: Most massive translation datasets are "idiom-poor," giving models few opportunities to learn these figurative nuances.

Methods

The authors explored two primary strategies using the T5-small architecture:

  1. Idiom-to-Idiom Translation (Direct): The model is fine-tuned to translate an English sentence containing an idiom directly into a German sentence containing an equivalent idiom. The researchers tested four variants of this, including "idiom-aware" versions where the specific idiom was appended to the input string (e.g., translate English to German: She’s not in the picture. idiom: to be in the picture).
  2. Idiom Paraphrasing and Translation (Pipeline): This is a two-step process.
  3. Step A: An English sentence with an idiom is "simplified" into a literal English paraphrase (e.g., "He can paddle his own canoe" $\rightarrow$ "He can be self-reliant").
  4. Step B: This simplified English is then translated into German using a standard MT model.

To evaluate these, they used both automatic metrics (BLEU, METEOR, and COMET) and a human survey where 53 participants rated the translations for fluency and meaning preservation.

Datasets

The study relied on two key datasets:

  • IdiomTranslationDS: A parallel corpus of ~3,500 English-German sentence pairs. After manual cleaning by German speakers to remove incorrect translations and noise, the authors were left with a smaller, high-quality set for fine-tuning.
  • PIE (Potential Idiomatic Expression) Dataset: Used for the paraphrasing task, containing over 5,000 English sentences with their corresponding literal paraphrases and idiom meanings.
  • TaPaCo: A corpus of sentential paraphrases used to pre-train the paraphrasing component of the pipeline approach.

Key Insights and Ideas

  • Contextual Embeddings are the Secret Sauce: Because T5 uses contextual embeddings, it doesn't just look at the word "ice"; it looks at the words around it. This allows the model to better distinguish between a bartender literally breaking ice and a person figuratively "breaking the ice."
  • The "Pipeline Tax": While the paraphrasing pipeline seems logical (simplify then translate), it suffers from error propagation. If the paraphraser makes a slight mistake in Step A, that error is magnified during the translation in Step B.
  • Direct Fine-tuning Wins: Surprisingly, even with a relatively small amount of idiom-specific data, the direct translation approach outperformed the complex pipeline. This suggests that T5 is remarkably efficient at picking up figurative mappings if given the right examples.
  • Task Framing Matters: The best results came from continuing to fine-tune T5 on the predefined "translate English to German" task using the idiom-specific data, rather than creating a brand-new custom task prefix.

Key Results

  • Automatic Evaluation: The direct fine-tuned model achieved the highest scores across the board. On the IdiomTranslationDS test set, it reached a BLEU of 0.151 and a METEOR of 0.498, significantly outperforming the baseline T5 model.
  • Human Evaluation:
    • Fluency: The direct translation model scored an average of 3.73/5, while the pipeline approach scored 3.65/5.
  • Meaning Preservation: Both models performed similarly, with the direct approach at 3.30/5 and the pipeline at 3.29/5.
  • Statistical Significance: A paired t-test confirmed that the improvements in translation quality using these fine-tuning methods were statistically significant compared to the baseline.

Limitations and Future Work

The authors identify several hurdles for the future:

  • Data Volume: The biggest limitation is the scarcity of parallel idiom data. Fine-tuning on only ~1,700 pairs is a "low-resource" scenario.
  • Language Scope: This study focused exclusively on English-to-German. Idiomatic logic varies wildly across language families (e.g., Asian or African languages).
  • Prompt-based Learning: The authors suggest that future work could treat idiom translation as a "prompt-based" task, potentially allowing models to leverage their massive pre-trained knowledge without needing large fine-tuning datasets.
  • Dataset Enrichment: They propose creating a "master dataset" that combines idiom detection, paraphrasing, and translation into a single multi-task learning framework.

Summary updated

Abstract

Idiomatic expressions (or idioms) are phrases where the meaning of the phrase cannot be determined from the meaning of the individual words in the expression. Translating idioms between languages is therefore a challenging task. Transformer models based on contextual embeddings have advanced the state-of-the-art across many domains in the field of natural language processing. While research using transformers has advanced both idiom detection as well as idiom disambiguation, idiom translation has not seen a similar advancement. In this work, we investigate two approaches to fine-tuning a pretrained Text-to-Text Transfer Transformer (T5) model to perform idiom translation from English to German. The first approach directly translates English idiom-containing sentences to German, while the second is underpinned by idiom paraphrasing, firstly paraphrasing English idiomatic expressions to their simplified English versions before translating them to German. Results of our evaluation show that each of the approaches is able to generate adequate translations.
BibTeX
@misc{santing2022food,
  author = {Lukas Santing and Ryan Sijstermans and Giacomo Anerdi and Pedro Jeuris and Marijn ten Thij and Riza Batista-Navarro},
  title = {{Food for Thought: How can we exploit contextual embeddings in the translation of idiomatic expressions?}},
  year = {2022},
  doi = {10.18653/v1/2022.flp-1.14},
}
Figure