Adaptation of Word-Level Benchmark Datasets for Relation-Level Metaphor Identification

Omnia Zayed, John P. McCrae, Paul BuitelaarACL 2020

TL;DR

This paper addresses the scarcity of suitable datasets for relation-level metaphor identification, a task crucial for deeper metaphor understanding. The authors adapt existing word-level metaphor datasets (VUAMC, TroFi) and enhance a relation-level dataset (TSV) to provide resources for this under-explored area, enabling more robust computational metaphor analysis.

What Problem Does This Paper Address?

Metaphor is a fundamental aspect of human language, allowing us to understand abstract concepts through concrete ones (e.g., "time is money"). Computational linguistics has made strides in identifying metaphors, but most existing work focuses on "word-level" identification – simply flagging a word as metaphorical. However, true understanding requires grasping the relation between the metaphorical term (vehicle) and the concept it represents (tenor). This "relation-level" identification is more challenging and requires datasets annotated with this richer information. Current relation-level datasets are either limited in scope or lack full sentence context, hindering the development of advanced models. This paper aims to bridge this gap by adapting and enhancing existing datasets to better support relation-level metaphor identification.

Methods

The core methodology involves adapting existing word-level metaphor datasets to a relation-level format. This is achieved through a semi-automatic process:

  1. Dataset Selection: The authors choose three widely used datasets: VU Amsterdam Metaphor Corpus (VUAMC), TroFi, and TSV.
  2. Grammatical Relation Extraction: They employ the Stanford dependency parser to extract specific grammatical relations that typically link the metaphorical vehicle (e.g., a verb or adjective) to its tenor (e.g., a direct object or noun). For verb-noun metaphors, they focus on dobj (direct object) relations. For adjective-noun metaphors, they focus on amod (adjectival modifier) relations.
  3. Gold Label Retrieval: The original word-level metaphor labels are retrieved for the identified relations.
  4. Quality Verification: A crucial step involves manual inspection and verification of the extracted relations and their assigned labels. This includes checking the syntactic validity of the dependency, ensuring the verb/adjective is indeed metaphorical due to its object, and confirming the expression is a genuine metaphor.
  5. TSV Dataset Enhancement: For the TSV dataset, which already focuses on adjective-noun relations but lacks full sentence context in its training set, the authors use the Twitter Search API to retrieve tweets containing the adjective-noun pairs. This provides contextual information for training.

Datasets

The paper focuses on adapting and enhancing the following datasets:

  • VU Amsterdam Metaphor Corpus (VUAMC): Originally annotated for word-level metaphor identification. The authors adapt it for verb-direct object relations.
  • TroFi Dataset: Also originally for word-level metaphor identification (specifically verbs). It's adapted for verb-direct object relations.
  • TSV Dataset: Already a relation-level dataset for adjective-noun relations, but with limited context in its training set. The authors enhance it by adding tweet contexts.

The paper publishes these adapted datasets to facilitate future research.

Key Insights and Ideas

  • Word-level vs. Relation-level: The paper clearly articulates the distinction, highlighting that relation-level analysis is more aligned with human comprehension by explicitly considering the tenor and the link between tenor and vehicle.
  • Dataset Limitations: Existing word-level datasets, while abundant, are not directly usable for relation-level tasks without significant effort. Relation-level datasets are often smaller or lack crucial contextual information.
  • Adaptation is Feasible: The paper demonstrates that existing word-level datasets can be effectively adapted for relation-level tasks using dependency parsing and manual verification.
  • Context is King (Especially for TSV): The enhancement of the TSV dataset with tweet contexts underscores the importance of surrounding text for understanding and identifying metaphors, particularly in user-generated content.
  • Annotation Quality Concerns: The authors raise concerns about the consistency and theoretical grounding of annotations in some widely used datasets (like VUAMC), suggesting that relation-level annotation might be more robust.

Key Results

The paper presents statistics on the adapted datasets:

  • Adapted VUAMC: 4,420 sentences in the training set and 1,398 in the test set, focusing on verb-direct object relations. Approximately 38-42% of instances are metaphors.
  • Adapted TroFi: 1,535 sentences, focusing on verb-direct object relations. Approximately 59-50% of instances are metaphors.
  • Enhanced TSV: 1,764 tweets added to the training set, providing context for adjective-noun relations.

A quality assessment showed high agreement (e.g., 98.52% for syntactic validity in TroFi, 98.5% for verb metaphoricity in VUAMC) for the adapted datasets, though some annotation disagreements were noted for VUAMC.

Limitations and Future Work

  • VUAMC Annotation Issues: The authors acknowledge that the original VUAMC annotations have quality and consistency issues, which were also flagged during their adaptation process.
  • Limited Relation Types: The current adaptation primarily focuses on verb-direct object and adjective-noun relations.
  • Future Work: The authors plan to expand the adapted VUAMC to include verb-subject (nsubj) and adjective-noun (amod) relations. They also aim to consolidate the adapted datasets into a single repository and explore weakly supervised approaches for further annotation consistency.

Summary updated

BibTeX
@inproceedings{zayed2020adaptation,
  author = {Omnia Zayed and John P. McCrae and Paul Buitelaar},
  title = {{Adaptation of Word-Level Benchmark Datasets for Relation-Level Metaphor Identification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}

Loading related content...

Figure