EPIE Dataset: A Corpus for Possible Idiomatic Expressions

Prateek Saxena, Soma Paul

TL;DR

The EPIE (English Possible Idiomatic Expressions) dataset provides a large-scale corpus of over 25,000 sentences covering 717 distinct idioms to help AI systems recognize non-compositional language. By distinguishing between "Static" and "Formal" (variable) idioms, the researchers provide a robust framework for sequence labeling that significantly outperforms previous small-scale datasets.

What Problem Does This Paper Address?

Idiomatic expressions (IEs) like "kick the bucket" or "keep an eye on" are a major hurdle for Natural Language Processing (NLP), particularly Machine Translation (MT). Because the meaning of an idiom is non-compositional—meaning it cannot be understood by simply adding up the definitions of its individual words—MT systems often produce nonsensical literal translations.

The core challenges are:

  1. Sparsity: Idioms appear far less frequently in training data than the literal usage of their constituent words.
  2. Lexical Variation: While some idioms are fixed, many undergo changes in tense, number, or even have words inserted (e.g., "keep a very close eye on").
  3. Dataset Size: Prior datasets were limited to roughly 60 candidate idioms, which is insufficient for training deep learning models to generalize across the vast landscape of human expression.

Methods

The authors developed a semi-automatic pipeline to build the EPIE Corpus:

  • Extraction via StringNet: They used StringNet, a massive resource of 2 billion hybrid n-grams, to find instances of idioms from the British National Corpus (BNC).
  • Pruning Techniques: To capture variations, they employed "Vertical Pruning" (generalizing lexemes, like replacing "his" with a generic [pron] tag) and "Horizontal Pruning" (connecting n-grams that differ by only one unit).
  • Categorization: They split idioms into two types:
  • Static: Fixed phrases that don't change (e.g., "as soon as possible"). These can be found via simple string matching.
  • Formal: Phrases that allow internal modifications (e.g., "bite the dust" becoming "bit the dust").
  • Sequence Labeling: For the more complex "Formal" idioms, the authors trained a BiLSTM-CRF (Bidirectional Long Short-Term Memory with Conditional Random Fields) model. This architecture is standard for identifying spans of text (like Named Entity Recognition) but was applied here to detect the boundaries of idiomatic expressions.

Datasets

  • EPIE (New): 25,206 sentences containing 717 idiomatic expressions. It is split into 21,891 sentences for 359 Static idioms and 3,135 sentences for 358 Formal idioms.
  • Evaluation Benchmarks: To prove the utility of EPIE, the authors tested their model on three independent datasets:
  • SemEval-2013 Task 5b: Both the "all words" and "lex sample" subsets.
  • PIE Corpus: A pre-existing corpus of potentially idiomatic expressions.
  • IMIL Dataset: Used as the initial source for the list of 2,000 common English idioms.

Key Insights and Ideas

  • The "Formal" vs. "Static" Divide: The most important insight is that not all idioms should be treated the same. Static idioms are a search problem; Formal idioms are a machine learning problem.
  • Span Detection vs. Disambiguation: The authors focus on "Possible" idiomatic expressions. Their goal is to identify the span of the phrase first. Once an AI knows where a potential idiom is, it is much easier to perform the second step: determining if the usage is literal or metaphorical.
  • Distribution Skew: The data showed that Static idioms have a much higher variance in frequency (Standard Deviation of 160) compared to Formal idioms (SD of 8.61). This suggests that Static idioms are "hit or miss" in common usage, while Formal idioms appear with more consistent, albeit lower, frequency.

Key Results

The BiLSTM-CRF model trained on the EPIE "Formal" dataset showed impressive generalization:

  • Internal Test Set: Achieved 0.98 Accuracy, 0.95 Precision, and 0.91 Recall.
  • Cross-Dataset Performance: On the SemEval "lex sample" (idiomatic usages), the model maintained a high 0.92 Accuracy.
  • PIE Corpus: The model achieved an 0.88 Accuracy on idiomatic usages, though it struggled more on "all usages" (0.69), likely due to the difficulty of distinguishing literal from idiomatic meanings in that specific dataset.

Limitations and Future Work

  • Manual Filtering: While the extraction was largely automated, the authors had to manually remove irrelevant patterns for Formal idioms to ensure data quality. This limits the speed at which the dataset can be expanded to tens of thousands of idioms.
  • Literal/Idiomatic Disambiguation: The current work focuses on identifying the presence of an idiomatic phrase (lexical occurrence). The authors acknowledge that the next step is to use these identified spans to improve Metaphor Detection—actually deciding if "bit the dust" refers to a fallen soldier or someone literally tasting soil.
  • MT Integration: While the paper mentions Machine Translation as a motivation, it does not yet implement the dataset into an MT pipeline to measure the actual BLEU score improvement.

Summary updated

Abstract

Idiomatic expressions have always been a bottleneck for language comprehension and natural language understanding, specifically for tasks like Machine Translation (MT). MT systems predominantly produce literal translations of idiomatic expressions as they do not exhibit generic and linguistically deterministic patterns which can be exploited for comprehension of the non-compositional meaning of the expressions. These expressions occur in parallel corpora used for training, but due to the comparatively high occurrences of the constituent words of idiomatic expressions in literal context, the idiomatic meaning gets overpowered by the compositional meaning of the expression. State of the art Metaphor Detection Systems are able to detect non-compositional usage at word level but miss out on idiosyncratic phrasal idiomatic expressions. This creates a dire need for a dataset with a wider coverage and higher occurrence of commonly occurring idiomatic expressions, the spans of which can be used for Metaphor Detection. With this in mind, we present our English Possible Idiomatic Expressions (EPIE) corpus containing 25,206 sentences labelled with lexical instances of 717 idiomatic expressions. These spans also cover literal usages for the given set of idiomatic expressions. We also present the utility of our dataset by using it to train a sequence labelling module and testing on three independent datasets with high accuracy, precision and recall scores.
BibTeX
@inproceedings{saxena2020epie,
  author = {Prateek Saxena and Soma Paul},
  title = {{EPIE Dataset: A Corpus for Possible Idiomatic Expressions}},
  booktitle = {TDS},
  year = {2020},
  url = {https://openreview.net/forum?id=UPR9W2K_gu},
}

Loading related content...

Figure