SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articles

Ana-Cristina Rogoz, Mihaela Găman, Radu Tudor IonescuACL 2021

TL;DR

This paper introduces SaRoCo, the first and largest corpus for detecting satirical news in Romanian, containing over 55,000 articles. They demonstrate that current state-of-the-art deep learning models struggle to achieve human-level performance on this task, highlighting a significant gap for future research.

What Problem Does This Paper Address?

Satire, a form of humorous criticism using irony and parody, is often disguised as legitimate news. Distinguishing satirical news from genuine reporting is crucial for understanding public discourse and preventing the spread of misinformation, especially as AI systems become more integrated into our lives. While satire detection has been explored in several languages, there was a notable lack of resources for Romanian. This paper aims to fill that gap by creating a dedicated Romanian corpus and establishing baseline performance for existing models.

Methods

The authors developed a novel dataset, SaRoCo, by collecting 55,608 Romanian news articles from various real and satirical sources. To ensure robust evaluation, they implemented a strict data split, ensuring that training and testing articles come from different sources, preventing models from overfitting to specific publication styles. They then evaluated two state-of-the-art deep learning models:

  • Fine-tuned Ro-BERT: A Romanian version of the BERT transformer model, adapted for the specific task of satire detection.
  • Character-level CNN: A Convolutional Neural Network that processes text at the character level, designed to capture nuanced linguistic patterns.

Additionally, they employed Grad-CAM to analyze the discriminative features learned by the CNN, aiming to understand why the model makes certain predictions and to ensure its decisions aren't based on superficial biases like named entities.

Datasets

  • SaRoCo: A newly introduced dataset comprising 55,608 Romanian news articles (27,628 satirical, 27,980 non-satirical). This corpus is split into training, validation, and test sets, with articles in each split originating from distinct sources.
  • Existing Corpora: The paper also provides a comparative analysis of SaRoCo against existing satire detection corpora in various languages (English, French, German, Spanish, Arabic, Turkish) to contextualize its scale and significance.

Key Insights and Ideas

  • Romanian Satire Detection is Challenging: The performance gap between human annotators and even the best deep learning models is substantial, indicating that Romanian satire is complex and difficult for machines to grasp.
  • Named Entity Removal is Crucial: The authors found that removing named entities (like people's names or organizations) is important to prevent models from relying on superficial clues and to ensure they focus on the linguistic style of satire. This also makes the task harder, even for humans.
  • Character-level CNNs Offer Interpretability: Using Grad-CAM on the character-level CNN allowed the researchers to visualize the features the model uses. They observed that the CNN tends to pick up on obvious cues like slang and popular sayings in satirical content, rather than deeper ironic structures.
  • Human vs. Machine Decision-Making: Both humans and models exhibit a similar bias: higher precision for satirical content and higher recall for regular news. This suggests a shared strategy of looking for positive indicators of satire, and defaulting to "regular" if those indicators are absent.

Key Results

  • Baseline Performance: The fine-tuned Ro-BERT model achieved a maximum accuracy of approximately 73% on the test set for full articles, while the character-level CNN reached around 69.6%.
  • Human Performance: In contrast, human annotators achieved an average accuracy of about 87% on a subset of headlines.
  • Headline vs. Full Article: Performance on headlines alone (Table 8) showed a larger gap between humans (87.35% accuracy) and models (Ro-BERT: 68%, Char-CNN: 65%). This suggests that full articles provide more context for detection.

Limitations and Future Work

  • Model Limitations: The authors acknowledge that current deep learning models struggle to capture the nuances of satire, often relying on surface-level features. The performance gap between models and humans is significant, leaving ample room for improvement.
  • Future Directions: The paper suggests that future research should focus on developing models that can better pick up on irony and exaggeration. Further exploration into more sophisticated feature extraction and model architectures is needed to close the performance gap. The SaRoCo corpus and baselines are made publicly available to encourage further research in this area.

Summary updated

BibTeX
@inproceedings{rogoz2021saroco,
  author = {Ana-Cristina Rogoz and Mihaela Găman and Radu Tudor Ionescu},
  title = {{SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure