‘... like a needle in a haystack”: Annotation and Classification of Comparative Statements

Pritha Majumdar, Franziska Pannach, Arianna Graciotti, Johan BosNAACL 2025

TL;DR

This paper tackles the challenge of automatically distinguishing between literal comparisons and figurative similes, a subtle but crucial distinction in natural language processing. They introduce a fine-grained annotation guideline and a publicly available dataset, demonstrating that large language models, particularly with few-shot prompting, can achieve high accuracy in this classification task.

What Problem Does This Paper Address?

Figurative language, like similes, is notoriously difficult for computers to understand. While comparisons are common in everyday language to highlight similarities or differences between entities, similes elevate this by drawing parallels between dissimilar entities, often to create a specific image or convey deeper meaning. The challenge lies in the subtle linguistic cues that differentiate a straightforward comparison (e.g., "He is as tall as his brother") from a simile (e.g., "He is as tall as the tower"). This distinction is vital for tasks like literary analysis, sentiment analysis, and even more nuanced text generation. Existing work has often focused on broader figurative language tasks like metaphor detection, leaving the specific problem of simile vs. literal comparison under-addressed, especially concerning the development of clear annotation guidelines and dedicated datasets.

Methods

The authors propose a novel, fine-grained annotation methodology to distinguish between literal comparisons and similes. This methodology is built upon a detailed analysis of comparative statements, considering seven key characteristics: nature (quantitative vs. qualitative), categories (semantic similarity of entities), feature matching (prominence of similarities vs. differences), symmetry (interchangeability of subject and object), salience (importance of shared features), broad unifying concept (meaningfulness of the comparison), and domain incongruence (distinct semantic domains of entities).

For automatic classification, they implemented a baseline Support Vector Machine (SVM) model. They also experimented with large language models (LLMs), specifically Llama-3.3-70b-instruct, using both zero-shot and few-shot prompting strategies. The few-shot approach involved providing the LLM with a few examples to guide its classification.

Datasets

The authors created a new, publicly available annotated dataset of 100 comparative statements. This dataset was derived from the Parallel Meaning Bank (PMB) and manually cleaned and filtered. The dataset is crucial for training and evaluating their models, as well as for future research in this area.

Key Insights and Ideas

  • The subtle art of simile: The core insight is that the distinction between comparison and simile hinges on more than just syntactic structure; it involves semantic categories, the degree of similarity, and the underlying conceptual connection.
  • Categories matter: A key differentiator is whether the compared entities belong to the same super-ordinate category. If they do, it's likely a comparison; if they are from distinct categories requiring a higher level of abstraction to connect, it's more likely a simile.
  • Feature prominence is key: Similes often highlight differences or less obvious similarities, whereas comparisons tend to emphasize prominent shared features.
  • LLMs can learn nuance: The research demonstrates that LLMs, especially when guided by few-shot examples, can learn to discern these subtle linguistic distinctions, outperforming traditional machine learning models.
  • Cultural context influences interpretation: Human annotators' cultural backgrounds can lead to differing interpretations of statements, highlighting the complexity of figurative language annotation.

Key Results

The authors achieved an accuracy of 82% with their best-performing model, the Llama-3.3-70b-instruct LLM using a few-shot prompting strategy. This significantly outperforms their baseline SVM model, which achieved 75% accuracy, and the zero-shot LLM approach, which achieved 72% accuracy. Their human annotation study revealed an inter-annotator agreement score of 0.62 (Cohen's Kappa), indicating a moderate level of agreement, with some disagreements stemming from stock similes and cultural interpretations.

Limitations and Future Work

The authors acknowledge that while their LLM achieved high accuracy, it sometimes made unexpected decisions, particularly in cases of unspecified subjects/objects or idiomatic expressions. They also note that cultural differences can impact annotation.

Future work includes developing a more fine-grained annotation guideline for similes in literature, aiming for language-agnostic guidelines, and extending this to other languages like Bengali. They also plan to conduct a quantitative and contrastive analysis to explore cultural narratives and values depicted in simile usage and human expression.

Summary updated

BibTeX
@inproceedings{majumdar2025like,
  author = {Pritha Majumdar and Franziska Pannach and Arianna Graciotti and Johan Bos},
  title = {{‘... like a needle in a haystack”: Annotation and Classification of Comparative Statements}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure