Abstract
Idioms represent a significant challenge for Natural Language Understanding (NLU) due to their non-compositional nature. This paper introduces ID10M, a framework for identifying idioms across 10 languages by reframing the problem as a sequence-labeling task. By leveraging a massive new multilingual dataset and a novel training approach, the system demonstrates the ability to detect figurative language even for previously unseen expressions.
The Challenge of Idiomatic Language
Idioms are often considered the "final boss" of NLU. When a speaker says they are "feeling under the weather," a human understands they are unwell. However, a standard AI might interpret the phrase literally, looking for meteorological anomalies. This occurs because idioms are non-compositional: their meaning cannot be derived from the sum of their individual parts.
Historically, idiom research has faced two major hurdles:
- English-centricity: The vast majority of datasets and models are limited to English.
- The "Pre-identification" Trap: Most existing systems require a potential idiom to be pre-identified before determining if its usage is literal or figurative. In real-world applications, an AI must be capable of discovering idioms within raw, unstructured text.
Methodology
The researchers developed a two-pronged approach to address these challenges: automatic data generation and sequence labeling.
Automatic "Silver" Data Generation
To facilitate large-scale training, the team generated "silver" data using Wiktionary for idiom discovery and WikiMatrix for sentence retrieval. To ensure high data quality, they employed a Dual-Encoder Discriminator. This component compares the vector representation of a phrase (e.g., "break the ice") with its surrounding context. A significant "distance" between the phrase and its context suggests idiomatic usage.
Sequence Labeling via BIO Tagging
Rather than treating idiom detection as a simple classification task for pre-selected phrases, the authors framed it as Named Entity Recognition (NER). They utilized a BIO (Beginning, Inside, Outside) tagging scheme:
- B-IDIOM: Marks the start of an idiomatic expression.
- I-IDIOM: Marks the continuation of the expression.
- O: Represents literal, non-idiomatic tokens.
This approach allows the model to automatically "highlight" idiomatic segments within a sentence. The final architecture combines a multilingual Transformer (mBERT) with a BiLSTM and a CRF (Conditional Random Field) layer to capture both semantic depth and sequential dependencies.
The ID10M Dataset
The paper introduces a comprehensive suite of datasets:
- Silver Training Set: Automatically generated data covering 10 languages: Chinese, Dutch, English, French, German, Italian, Japanese, Polish, Portuguese, and Spanish.
- Gold Evaluation Benchmark: A manually curated test set for 4 languages (English, German, Italian, and Spanish). This benchmark includes both "seen" idioms (present in training) and "unseen" idioms to evaluate the model's generalization capabilities.
Key Results and Insights
The experimental results yielded several critical insights:
- Generalization to Unseen Idioms: The model successfully identified idioms it had never encountered during training. This suggests the system learns the contextual signature of figurative language rather than just memorizing a static list of phrases.
- Data Quality Matters: Using the Dual-Encoder to filter the silver training data resulted in a performance boost of approximately 4.3 F1 points, underscoring the importance of data validation.
- Model Performance: The system achieved an average macro-F1 score of 76.1 across the four gold-standard languages. While performance on "seen" idioms was exceptional (91.5 F1), it maintained a respectable score of 59.6 F1 on entirely new expressions.
Limitations and Future Directions
A notable limitation is the system bias toward frequent usages. If a phrase is used idiomatically in 90% of the training data, the model may struggle with the 10% of cases where it is used literally. For instance, the phrase "kick the bucket" might be misidentified in a literal context involving a janitor and a cleaning pail.
Future work will focus on expanding the framework to more languages and integrating idiom-awareness into downstream tasks such as Machine Translation and Question Answering to improve overall NLU robustness.
Summary updated