To Leave No Stone Unturned: Annotating Verbal Idioms in the Parallel Meaning Bank

Rafael Ehren, Kilian Evang, Laura KallmeyerCOLING 2024

TL;DR

This paper tackles the challenge of accurately annotating verbal idioms in the Parallel Meaning Bank (PMB), a semantic resource. By developing new guidelines and an annotation process, the authors significantly improve the representation of these tricky expressions, making them more accessible for downstream NLP tasks like semantic parsing.

What Problem Does This Paper Address?

Semantic annotation of natural language is crucial for building intelligent systems, but multiword expressions (MWEs), especially verbal idioms (VIDs), pose a significant hurdle. VIDs, like "spill the beans," often have meanings that cannot be deduced from their individual words, making them difficult to represent in lexicalized semantic frameworks. Existing resources like the PMB have historically underrepresented or inadequately annotated VIDs, often treating them literally. This paper aims to bridge this gap by creating a more comprehensive and accurate semantic annotation of German verbal idioms within the PMB.

Methods

The authors employed a multi-step approach:

  1. Candidate Extraction: They first identified potential German verbal idiom candidates by leveraging the Redensarten-Index, a large online dictionary of idioms and proverbs. This initial list was then filtered to focus specifically on verbal expressions.
  2. Sentence Retrieval: Using a parsing-based extraction method, they retrieved sentences from the PMB that contained these identified idiom lemmata, ensuring the correct dependency relations were present to avoid accidental co-occurrences.
  3. Annotation Process: Three linguistically trained native speakers annotated the extracted sentences. This involved:
  4. Filtering False Positives: Removing expressions that didn't fit the definition of verbal idioms (e.g., proverbs, light-verb constructions).
  5. Idiomaticity Judgment: Determining if an expression was idiomatic, probably idiomatic, probably literal, or literal.
  6. Decomposability Assessment: Classifying idioms as decomposable (parts map to meanings) or non-decomposable (meaning is holistic). They also introduced "mixed" and "LVC" (light-verb construction) categories to handle nuanced cases.
  7. Semantic Annotation: Mapping the idiomatic meaning to WordNet senses and assigning VerbNet semantic roles to the idiom's arguments.

Datasets

  • Parallel Meaning Bank (PMB): The primary dataset used for annotation. The paper focuses on the German partition of the PMB.
  • Redensarten-Index: Used as a source for identifying candidate German verbal idioms.
  • Potentially Idiomatic Expressions (PIEs): As a byproduct, the authors created a dataset of PIEs, which includes both literal and idiomatic readings of expressions.

Key Insights and Ideas

  • Decomposability is Key: The distinction between decomposable and non-decomposable idioms is crucial for semantic annotation. Non-decomposable idioms are significantly more challenging and often receive literal treatments in existing resources.
  • Nuance in Decomposability: The binary distinction of decomposability isn't always sufficient. The introduction of "mixed" and "LVC" categories acknowledges that some expressions have intermediate properties or are semantically bleached.
  • Word Sense Disambiguation is Hard: Mapping German idioms to English WordNet senses is a significant challenge, often leading to multiple plausible options or no suitable sense at all. This highlights the difficulty of cross-lingual semantic annotation.
  • Prioritizing Decomposition: Even when a very suitable holistic sense is available, the authors chose to prioritize a decomposable analysis if possible, leading to potentially less "elegant" but more granular semantic representations.
  • Collocations vs. Idioms: The distinction between "idioms of encoding" (collocations) and "idioms of decoding" (true idioms) is important for defining the scope of annotation, though it can be difficult to draw a clear line.

Key Results

  • High Inter-Annotator Agreement: Despite the complexity, the authors achieved encouraging inter-annotator agreement scores, particularly for head selection and argument identification (κ > .98). Agreement was weaker for head sense classification (κ = .5862), reflecting the challenges in word sense disambiguation.
  • Majority of Idioms are Non-Decomposable: Among the annotated idioms, a large majority were classified as non-decomposable.
  • Significant False Positives: The initial extraction process yielded a substantial number of false positives, necessitating a thorough filtering step during annotation.
  • Improved PMB Coverage: The work aims to integrate these new annotations into the PMB, moving towards a "gold standard" for sentences containing idioms.

Limitations and Future Work

  • Manual Correction Needed: While the process generates "silver" status documents, manual checking is still required for "gold" status.
  • Integrating Annotations: The next step is to integrate the new annotations into the PMB's dynamic annotation workflow, which involves adding new rules to the Boxer component.
  • English Idiom Annotation: An analogous annotation project for English idioms in the PMB is underway.
  • Handling Ambiguity: The paper acknowledges the ongoing challenge of word sense disambiguation and the need for robust methods to handle multiple plausible senses.
  • Defining Collocations: Further work is needed to establish a sharper criterion for distinguishing idioms from collocations.

Summary updated

BibTeX
@inproceedings{ehren2024leave,
  author = {Rafael Ehren and Kilian Evang and Laura Kallmeyer},
  title = {{To Leave No Stone Unturned: Annotating Verbal Idioms in the Parallel Meaning Bank}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure