LT3 at SemEval-2022 Task 6: Fuzzy-Rough Nearest Neighbor Classification for Sarcasm Detection

Olha Kaminska, Chris Cornelis, Veronique HosteNAACL 2022

TL;DR

This paper presents a novel approach to sarcasm detection using a fuzzy-rough nearest neighbor (FRNN) classifier combined with a powerful pre-trained language model (roBERTa) for text embedding. The system achieved a respectable 9th place in the SemEval-2022 "iSarcasmEval" competition, demonstrating the effectiveness of combining explainable classification methods with state-of-the-art deep learning embeddings.

What Problem Does This Paper Address?

Sarcasm detection is a notoriously challenging task in Natural Language Processing (NLP). Unlike sentiment analysis, sarcasm often involves a disconnect between the literal meaning of words and the intended message, making it difficult even for humans to identify. This ambiguity is further complicated by the fact that sarcasm can manifest in various ways, often relying on context, tone, or even non-verbal cues that are absent in text alone. This makes creating accurate and reliable datasets for sarcasm detection a significant hurdle. The SemEval-2022 Task 6, "iSarcasmEval," aimed to address this by providing a dataset where authors labeled their own text for sarcasm, aiming to reduce subjective bias. The paper tackles the binary classification subtask for English, determining whether a given text is sarcastic or not.

Methods

The core of the LT3 team's approach lies in combining two key components:

  • Text Embedding: The authors explored several text embedding techniques to convert raw text into numerical vectors. They found that the Twitter-roBERTa-base for Irony Detection model (a fine-tuned version of RoBERTa) performed exceptionally well. This model leverages the power of transformer architectures to capture contextual nuances in text.
  • Fuzzy-Rough Nearest Neighbor (FRNN) Classifier: Instead of relying solely on black-box deep learning models, the team opted for a more explainable instance-based learning method, the FRNN classifier. This method, based on fuzzy set theory and rough set theory, uses lower and upper approximations of data to classify new instances. They specifically employed an extension of FRNN that incorporates Ordered Weighted Average (OWA) operators. OWA operators allow for flexible aggregation of neighbor information, making the classifier more robust. The FRNN classifier uses the cosine similarity to measure the relationship between text embedding vectors.

The overall architecture involved preprocessing the text, embedding it using the chosen method, and then feeding these embeddings into the OWA-FRNN classifier. They also experimented with an ensemble of FRNN models, but found that a single, well-tuned model performed best.

Datasets

The primary dataset used in this work is the iSarcasmEval dataset for SemEval-2022 Task 6. This dataset was specifically designed for sarcasm detection and includes:

  • Text of the utterance.
  • A binary label indicating whether the text is sarcastic.
  • A non-ironical rephrase of the text with an explanation of why it's sarcastic (used for subtask C, not in this paper's experiments).
  • Binary classes for six types of sarcasm (subtask B, not used here).

A key characteristic of the iSarcasmEval dataset for subtask A (English) is its imbalanced nature, with the non-irony class being three times larger than the irony class (2,601 vs. 867 instances). The authors used a 5-fold cross-validation on the training data for hyperparameter tuning and model selection, and then evaluated the final model on the held-out test data.

Key Insights and Ideas

  • Explainability Matters: The paper highlights the value of using more interpretable models like FRNN, even when powerful black-box models like RoBERTa are available. The ability to trace back which training instances influenced a prediction is crucial for understanding model behavior and debugging.
  • Pre-trained Embeddings are Powerful: The success of the Twitter-roBERTa-base model underscores the significant advancements in NLP thanks to large pre-trained language models. These models capture rich semantic and contextual information that is vital for nuanced tasks like sarcasm detection.
  • Ensembles Aren't Always Better: Contrary to common practice, the authors found that combining multiple models in an ensemble did not improve performance. This suggests that the top-performing single model (roBERTa + FRNN) was already highly effective, and adding weaker models could dilute its performance.
  • Topic and Keywords are Important: The error analysis revealed that common topics and keywords play a significant role in how the FRNN classifier groups instances. This provides a qualitative understanding of the model's decision-making process.

Key Results

  • The best performing setup, consisting of no text preprocessing, Twitter-roBERTa-base for irony detection embeddings, and an OWA-FRNN classifier with k=5 neighbors, achieved an F1-score of 0.4242 on the test data.
  • This result secured the LT3 team 9th place in the SemEval-2022 iSarcasmEval competition leaderboard for the English subtask A.
  • They also achieved competitive rankings on other metrics: 7th place for averaged F-score (0.6552), 8th place for accuracy (0.8100), and 6th place for precision (0.6422).
  • Interestingly, while cross-validation suggested certain hyperparameter settings (like k=17) were suboptimal, these settings performed significantly better on the actual test data, indicating potential differences between the cross-validation split and the final test set.

Limitations and Future Work

  • Dataset Imbalance: The inherent imbalance in the iSarcasmEval dataset (more non-sarcastic than sarcastic examples) poses a challenge for any classification model.
  • Subjectivity of Sarcasm: The paper acknowledges the inherent difficulty and subjectivity in labeling sarcasm, which can impact dataset quality and model performance.
  • Future Work: The authors suggest several avenues for improvement, including:
  • Exploring additional text preprocessing techniques.
  • Fine-tuning the roBERTa-based model on larger or more diverse datasets.
  • Investigating other fuzzy-rough classification extensions or ensemble strategies.

Summary updated

BibTeX
@inproceedings{kaminska2022lt,
  author = {Olha Kaminska and Chris Cornelis and Veronique Hoste},
  title = {{LT3 at SemEval-2022 Task 6: Fuzzy-Rough Nearest Neighbor Classification for Sarcasm Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure