EPIE Dataset: A Corpus For Possible Idiomatic Expressions

Prateek Saxena, Soma Paul

Abstract

Idiomatic expressions have always been a bottleneck for language comprehension and natural language understanding, specifically for tasks like Machine Translation (MT). MT systems predominantly produce literal translations of idiomatic expressions as they do not exhibit generic and linguistically deterministic patterns which can be exploited for comprehension of the non-compositional meaning of the expressions. These expressions occur in parallel corpora used for training, but due to the comparatively high occurrences of the constituent words of idiomatic expressions in literal context, the idiomatic meaning gets overpowered by the compositional meaning of the expression. State of the art Metaphor Detection Systems are able to detect non-compositional usage at word level but miss out on idiosyncratic phrasal idiomatic expressions. This creates a dire need for a dataset with a wider coverage and higher occurrence of commonly occurring idiomatic expressions, the spans of which can be used for Metaphor Detection. With this in mind, we present our English Possible Idiomatic Expressions (EPIE) corpus containing 25,206 sentences labelled with lexical instances of 717 idiomatic expressions. These spans also cover literal usages for the given set of idiomatic expressions. We also present the utility of our dataset by using it to train a sequence labelling module and testing on three independent datasets with high accuracy, precision and recall scores.

Introduction: The "Idiom Bottleneck"

Idiomatic expressions (IEs) such as "kick the bucket" or "keep an eye on" represent a significant challenge for Natural Language Processing (NLP). The primary issue is their non-compositional nature: the meaning of the whole cannot be derived from the sum of its parts. Consequently, Machine Translation (MT) systems often default to literal, nonsensical translations.

The "idiom bottleneck" is characterized by three main factors:

  1. Sparsity: Idioms appear much less frequently in training corpora than the literal usages of their component words.
  2. Lexical Variation: While some idioms are fixed, many are "formal" and undergo changes in tense, number, or internal modification (e.g., "keep a very close eye on").
  3. Dataset Limitations: Existing datasets are often too small (covering only ~60 idioms) to effectively train deep learning models for broad generalization.

Methodology and Corpus Construction

To address these challenges, we developed a semi-automatic pipeline to construct the EPIE Corpus. The process began with extraction using StringNet, a resource containing 2 billion hybrid n-grams from the British National Corpus (BNC).

To handle lexical variation, we applied two pruning techniques:

  • Vertical Pruning: Generalizing lexemes (e.g., replacing specific pronouns with a generic [pron] tag).
  • Horizontal Pruning: Connecting n-grams that differ by only a single unit to capture structural variations.
Extraction steps and corpus statistics for the EPIE dataset.

The resulting corpus categorizes idioms into two types:

  • Static: Fixed phrases (e.g., "as soon as possible") identifiable via string matching.
  • Formal: Phrases allowing internal modifications (e.g., "bite the dust" vs. "bit the dust").

For the complex "Formal" idioms, we implemented a BiLSTM-CRF (Bidirectional Long Short-Term Memory with Conditional Random Fields) architecture. This model is designed for sequence labeling, specifically to identify the boundaries (spans) of potential idiomatic expressions.

Results and Discussion

The BiLSTM-CRF model trained on the EPIE "Formal" dataset demonstrated strong generalization capabilities across diverse test sets, including the SemEval-2013 Task 5b and the PIE Corpus.

Performance of the BiLSTM-CRF model across various test datasets.

Key findings include:

  • High Precision and Recall: On our internal test set, the model achieved an accuracy of 0.98 and a precision of 0.95.
  • Cross-Dataset Robustness: The model maintained a 0.92 accuracy on the SemEval "lex sample" idiomatic usages.
  • Span Detection vs. Disambiguation: Our approach focuses on identifying "Possible" idiomatic expressions. By first identifying the span of a potential idiom, the subsequent task of disambiguating literal vs. metaphorical usage becomes significantly more tractable.
  • Distributional Insights: Data analysis revealed that Static idioms have much higher frequency variance compared to Formal idioms, suggesting that while Static idioms are common, their usage is highly concentrated in specific phrases.

Conclusion and Future Work

The EPIE dataset provides a robust foundation for improving idiom recognition in NLP. While the current extraction process involves some manual filtering to ensure quality, it significantly expands the scope of available training data. Future work will focus on integrating these identified spans into Metaphor Detection systems to decide if a phrase is being used literally or figuratively. Additionally, we aim to integrate this dataset into MT pipelines to quantify improvements in translation quality.

Summary updated

Abstract

Idiomatic expressions have always been a bottleneck for language comprehension and natural language understanding, specifically for tasks like Machine Translation(MT). MT systems predominantly produce literal translations of idiomatic expressions as they do not exhibit generic and linguistically deterministic patterns which can be exploited for comprehension of the non-compositional meaning of the expressions. These expressions occur in parallel corpora used for training, but due to the comparatively high occurrences of the constituent words of idiomatic expressions in literal context, the idiomatic meaning gets overpowered by the compositional meaning of the expression. State of the art Metaphor Detection Systems are able to detect non-compositional usage at word level but miss out on idiosyncratic phrasal idiomatic expressions. This creates a dire need for a dataset with a wider coverage and higher occurrence of commonly occurring idiomatic expressions, the spans of which can be used for Metaphor Detection. With this in mind, we present our English Possible Idiomatic Expressions(EPIE) corpus containing 25206 sentences labelled with lexical instances of 717 idiomatic expressions. These spans also cover literal usages for the given set of idiomatic expressions. We also present the utility of our dataset by using it to train a sequence labelling module and testing on three independent datasets with high accuracy, precision and recall scores.
BibTeX
@inproceedings{saxena2020epie,
  author = {Prateek Saxena and Soma Paul},
  title = {{EPIE Dataset: A Corpus For Possible Idiomatic Expressions}},
  booktitle = {TDS},
  year = {2020},
  url = {https://openreview.net/forum?id=UPR9W2K_gu},
}
Figure