ID10M: Idiom Identification in 10 Languages

Simone Tedeschi, Federico Martelli, Roberto NavigliNAACL 2022 · DOI

Abstract

Idioms represent a significant challenge for Natural Language Understanding (NLU) due to their non-compositional nature. This paper introduces ID10M, a framework for identifying idioms across 10 languages by reframing the problem as a sequence-labeling task. By leveraging a massive new multilingual dataset and a novel training approach, the system demonstrates the ability to detect figurative language even for previously unseen expressions.

The Challenge of Idiomatic Language

Idioms are often considered the "final boss" of NLU. When a speaker says they are "feeling under the weather," a human understands they are unwell. However, a standard AI might interpret the phrase literally, looking for meteorological anomalies. This occurs because idioms are non-compositional: their meaning cannot be derived from the sum of their individual parts.

Historically, idiom research has faced two major hurdles:

  1. English-centricity: The vast majority of datasets and models are limited to English.
  2. The "Pre-identification" Trap: Most existing systems require a potential idiom to be pre-identified before determining if its usage is literal or figurative. In real-world applications, an AI must be capable of discovering idioms within raw, unstructured text.

Methodology

The researchers developed a two-pronged approach to address these challenges: automatic data generation and sequence labeling.

Automatic "Silver" Data Generation

To facilitate large-scale training, the team generated "silver" data using Wiktionary for idiom discovery and WikiMatrix for sentence retrieval. To ensure high data quality, they employed a Dual-Encoder Discriminator. This component compares the vector representation of a phrase (e.g., "break the ice") with its surrounding context. A significant "distance" between the phrase and its context suggests idiomatic usage.

The Dual-Encoder Discriminator architecture used to validate idiomatic expressions in context.

Sequence Labeling via BIO Tagging

Rather than treating idiom detection as a simple classification task for pre-selected phrases, the authors framed it as Named Entity Recognition (NER). They utilized a BIO (Beginning, Inside, Outside) tagging scheme:

  • B-IDIOM: Marks the start of an idiomatic expression.
  • I-IDIOM: Marks the continuation of the expression.
  • O: Represents literal, non-idiomatic tokens.
An example of the BIO tagging scheme applied to the idiom "bite the bullet."

This approach allows the model to automatically "highlight" idiomatic segments within a sentence. The final architecture combines a multilingual Transformer (mBERT) with a BiLSTM and a CRF (Conditional Random Field) layer to capture both semantic depth and sequential dependencies.

The ID10M Dataset

The paper introduces a comprehensive suite of datasets:

  • Silver Training Set: Automatically generated data covering 10 languages: Chinese, Dutch, English, French, German, Italian, Japanese, Polish, Portuguese, and Spanish.
  • Gold Evaluation Benchmark: A manually curated test set for 4 languages (English, German, Italian, and Spanish). This benchmark includes both "seen" idioms (present in training) and "unseen" idioms to evaluate the model's generalization capabilities.

Key Results and Insights

The experimental results yielded several critical insights:

  • Generalization to Unseen Idioms: The model successfully identified idioms it had never encountered during training. This suggests the system learns the contextual signature of figurative language rather than just memorizing a static list of phrases.
  • Data Quality Matters: Using the Dual-Encoder to filter the silver training data resulted in a performance boost of approximately 4.3 F1 points, underscoring the importance of data validation.
  • Model Performance: The system achieved an average macro-F1 score of 76.1 across the four gold-standard languages. While performance on "seen" idioms was exceptional (91.5 F1), it maintained a respectable score of 59.6 F1 on entirely new expressions.

Limitations and Future Directions

A notable limitation is the system bias toward frequent usages. If a phrase is used idiomatically in 90% of the training data, the model may struggle with the 10% of cases where it is used literally. For instance, the phrase "kick the bucket" might be misidentified in a literal context involving a janitor and a cleaning pail.

Future work will focus on expanding the framework to more languages and integrating idiom-awareness into downstream tasks such as Machine Translation and Question Answering to improve overall NLU robustness.

Summary updated

BibTeX
@inproceedings{tedeschi2022id,
  author = {Simone Tedeschi and Federico Martelli and Roberto Navigli},
  title = {{ID10M: Idiom Identification in 10 Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.208},
}
Figure