Irony Detection in Hebrew Documents: A Novel Dataset and an Evaluation of Neural Classification Methods

Avi Shmidman, Elda Weizman, Avishay GerczukNAACL 2025

TL;DR

This paper introduces the first annotated dataset for detecting irony in Hebrew documents, specifically focusing on single words enclosed in quotation marks. They then use this dataset to evaluate various BERT-based neural network models, finding that while models can achieve good performance, distinguishing irony from other uses of quotation marks remains challenging.

What Problem Does This Paper Address?

Irony is a complex linguistic phenomenon that relies heavily on context and subtle cues for interpretation. While research on irony detection has advanced significantly, particularly in English, there's a notable gap in resources and studies for other languages, including Hebrew. Specifically, this paper addresses the challenge of identifying irony when it's signaled by single words enclosed in quotation marks. These "scare quotes" can be used for various purposes beyond irony, such as naming or highlighting peculiar word choices, making accurate classification difficult. The authors aim to create a dataset and evaluate machine learning models to distinguish these nuanced uses of quotation marks in Hebrew.

Methods

The core of the methodology involves training and evaluating neural classification models, specifically BERT-based architectures, on a newly created dataset. The authors experiment with different configurations to understand which features are most effective for irony detection.

  • Model Architecture: They utilize DictaBERT, a state-of-the-art BERT model pre-trained on modern Hebrew.
  • Classification Task: The goal is to classify instances of words in quotation marks into three categories: "Naming," "Lexical Peculiarity," and "Irony."
  • Feature Engineering: The experiments progressively add information to the model:
  • Masked Target Word: The model only sees the surrounding context, not the target word itself, to assess the importance of co-text.
  • Unmasked Target Word: The target word is included, allowing the model to leverage its specific meaning.
  • CLS Token Embedding: The embedding of the special [CLS] token, which represents the overall sentence meaning, is added.
  • Extended Co-text: The model is provided with preceding sentences within the same paragraph.
  • Redundancy Information: Features like the number of quoted words in a paragraph, paragraph length, and target word frequency are incorporated.
  • Evaluation: Performance is measured using precision, recall, and F1 scores for each category, often through 10-fold cross-validation. They also conduct binary classification experiments (e.g., Irony vs. Other) to isolate performance on specific categories.

Datasets

The paper introduces a novel annotated dataset for Hebrew irony detection.

  • Source: The dataset comprises op-ed articles from major Israeli news platforms, collected in 2019-2020.
  • Collection Methods: Data was gathered through automated crawling and manual collection.
  • Annotation: Three pragmatics experts annotated each instance of a single word in quotation marks within its full article context. They distinguished between "Naming," "Lexical Peculiarity," and "Irony."
  • Size: The dataset contains over 4000 instances (4,190 total), with a distribution of 45.1% Naming, 23.4% Lexical Peculiarity, and 31.5% Irony.
  • Significance: This is the first dataset of its kind for Hebrew, addressing a specific linguistic phenomenon that has not been systematically studied in this language.

Key Insights and Ideas

  • The Nuance of Quotation Marks: The study highlights that quotation marks are not a monolithic indicator of irony. They serve distinct pragmatic functions, including naming entities, signaling awareness of unusual word choices (lexical peculiarity), and conveying irony.
  • Context is King (Mostly): While the target word itself is crucial for accurate classification, the surrounding sentence context (co-text) provides significant clues for irony detection, even without seeing the target word.
  • The Power of the [CLS] Token: Incorporating the [CLS] token embedding, which summarizes the sentence's overall meaning, significantly boosted the model's performance, suggesting that a holistic sentence understanding is vital.
  • Extended Context Doesn't Always Help: Surprisingly, providing more extensive co-text (multiple preceding sentences) did not improve irony detection and even slightly decreased performance. This might indicate that models trained on single sentences struggle with longer contexts, or that the relevant cues are more localized.
  • Redundancy is Not Necessarily Helpful: Adding features related to redundancy (e.g., multiple quotes in a paragraph) also failed to improve irony detection, suggesting that the model's current architecture doesn't effectively leverage these signals.
  • Lexical Peculiarity is Tricky: Distinguishing "Lexical Peculiarity" from "Irony" proved to be the most challenging task for the neural network, likely due to their shared characteristics of meta-linguistic awareness and mild distanciation.

Key Results

  • Overall Performance: The best-performing model (unmasked target word + CLS embedding) achieved an F1 score of .825 for irony detection in a three-class setting.
  • Impact of Target Word: Unmasking the target word significantly improved performance across all categories compared to using only co-text (F1 for Irony increased from .780 to .818).
  • Benefit of [CLS] Token: Adding the [CLS] token embedding further boosted the F1 score for Irony to .825.
  • Binary Classification:
    • Irony vs. Other: Achieved an F1 score of .824 for Irony.
  • Naming vs. Other: Achieved very high F1 scores of .906 and .960 for Naming and Not Naming, respectively, indicating this category is relatively easy to identify.
  • Lexical Peculiarity vs. Other: Showed the lowest performance, with an F1 score of .529 for Lexical Peculiarity, highlighting its difficulty.

Limitations and Future Work

  • Lexical Peculiarity Challenge: The authors acknowledge that distinguishing "Lexical Peculiarity" from "Irony" remains a significant challenge. Future work could involve refining the analysis of this category, potentially by focusing on the specific lexical choices themselves.
  • Co-text Limitations: While the [CLS] token helped, the limited benefit of extended co-text suggests that current models might not fully leverage complex contextual information. Further research into how models handle multi-sentence contexts is warranted.
  • Model Perfection: The authors state that while the classifier can alert readers to potential irony ("red flag"), it is "not yet perfect" and improvements are desirable before deployment.
  • Future Directions: The authors plan to explore the lexical specificity of peculiar lexical choices and conduct further experiments to test their characterization of co-textual support for irony. They also suggest that the dataset can be used to promote future research and benchmarking in Hebrew irony detection.

Summary updated

BibTeX
@inproceedings{shmidman2025irony,
  author = {Avi Shmidman and Elda Weizman and Avishay Gerczuk},
  title = {{Irony Detection in Hebrew Documents: A Novel Dataset and an Evaluation of Neural Classification Methods}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure