Evaluating the Development of Linguistic Metaphor Annotation in Mexican Spanish Popular Science Tweets

Alec Sánchez-Montero, Gemma Bel-Enguix, Sergio-Luis Ojeda-TruebaNAACL 2024

TL;DR

This paper tackles the challenge of automatically detecting linguistic metaphors in Mexican Spanish popular science tweets, a novel and underexplored area. Despite using a rigorous annotation methodology, the study found surprisingly low inter-annotator agreement, highlighting the inherent subjectivity and complexity of metaphor identification, even among native speakers.

What Problem Does This Paper Address?

The automatic processing of metaphors in natural language has been a growing area of research within Artificial Intelligence and Natural Language Processing (NLP) for decades. However, most of this work has focused on English, leaving a significant gap for other languages, particularly Spanish. This paper specifically addresses the lack of resources and research for metaphor detection in Mexican Spanish popular science tweets. This is a crucial area because popular science communication often relies on metaphors to make complex scientific concepts accessible to the public, and understanding these metaphors computationally is key to better analyzing and processing this type of discourse. The authors aim to lay the groundwork for creating a robust dataset for machine learning in this domain.

Methods

The core of this research involves a manual annotation process for a corpus of tweets. The authors adapted the Metaphor Identification Procedure Vrije Universiteit (MIPVU), a well-established methodology for detecting linguistic metaphors in authentic texts. MIPVU identifies metaphorically used words (MRWs) and categorizes them into direct (DM), indirect (IM), and personification (PM) types.

For this study, the authors defined specific labels for each type:

  • Direct Metaphor (DM): Characterized by a source unit, target unit, and a signal/cue.
  • Indirect Metaphor (IM): An implicit comparison with only a source unit, where the target unit is elided.
  • Personification (PM): Attributing human or animate semantic features to inanimate or abstract objects.

The annotation was performed by four native Mexican Spanish-speaking undergraduate linguistics students using the Argilla platform. The corpus was split into two halves, with each half annotated by a pair of annotators to ensure balanced coverage and consistency.

Datasets

The primary dataset used in this study is a newly annotated corpus of 3733 Mexican Spanish popular science tweets. These tweets were collected from 19 science communicators based in Mexico, published between January 2020 and May 2023, and extracted via the X API. The corpus covers a wide range of scientific topics. This dataset is novel in its focus on Mexican Spanish and popular science discourse.

Key Insights and Ideas

  • The inherent subjectivity of metaphor: Even with clear guidelines and trained annotators, identifying metaphors is a subjective task. This is evidenced by the low inter-annotator agreement, suggesting that human intuition about metaphors can lead to divergent interpretations.
  • The complexity of indirect metaphors: The study found a notable predominance of indirect metaphors (IMs) in the exact matches, which also aligns with the general frequency of IMs in the corpus. This might be partly due to IMs requiring fewer labels for annotation compared to direct metaphors.
  • The need for iterative refinement: The authors emphasize that corpus annotation is an iterative process. The current findings highlight the necessity of refining annotation guidelines to reduce ambiguity and improve consistency for future annotation cycles.

Key Results

The study found low inter-annotator agreement, as measured by Cohen's kappa, which was only slightly above chance levels (0.16 for the first half and 0.17 for the second half). While the percentage of agreement was higher (74.27% and 63.63% respectively), the kappa scores indicate a "slight" agreement, considered a significant finding given the rigorous methodology.

Specifically, in terms of exact matches for metaphorical tweets:

  • In the first half of the corpus, only 12 out of 82 common metaphorical tweets matched exactly.
  • In the second half, only 27 out of 184 common metaphorical tweets showed complete agreement.
  • Overall, exact matches constituted 14.6% of the total metaphorical tweets.
  • Among these exact matches, Indirect Metaphors (IMs) were the most prevalent, accounting for 72.5%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Annotator inexperience: The annotators' lack of extensive experience in explicitly identifying metaphors may have contributed to interpretation errors.
  • Subjectivity of interpretation: Despite linguistic criteria, human interpretation of metaphorical expressions remains inherently subjective.

Future work will focus on:

  • Reworking the annotation guide: Developing a new version with clearer and more defined parameters to minimize ambiguity.
  • Establishing a Gold Standard dataset: Creating a definitive dataset of metaphorical tweets through further iterative annotation cycles.
  • Improving quality and reliability: Continuously enhancing the annotated dataset to ensure its effectiveness for automatic metaphor detection in Mexican Spanish.

Summary updated

BibTeX
@inproceedings{sanchezmontero2024evaluating,
  author = {Alec Sánchez-Montero and Gemma Bel-Enguix and Sergio-Luis Ojeda-Trueba},
  title = {{Evaluating the Development of Linguistic Metaphor Annotation in Mexican Spanish Popular Science Tweets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure