MLModeler5 at SemEval-2023 Task 3: Detecting the Category and the Framing Techniques in Online News in a Multi-lingual Setup

Arjun Khanchandani, Nitansh Jain, Jatin BediACL 2023

TL;DR

This paper presents MLModeler5's approach to SemEval-2023 Task 3, focusing on automatically detecting news genre and framing techniques across multiple languages. They leverage pre-trained BERT models (RoBERTa and ALBERT) and explore data augmentation and class weighting strategies to improve performance on this challenging multi-lingual task.

What Problem Does This Paper Address?

In today's information-saturated world, understanding how news is presented and how it influences public perception is crucial. This paper tackles the challenge of automatically analyzing online news content to identify its genre (e.g., opinion, report, satire) and the specific framing techniques used to shape narratives. This is particularly important for combating misinformation and propaganda, and for gaining a deeper understanding of media bias across different languages and cultures. The task aims to develop computational tools that can perform this analysis at scale, moving beyond manual human interpretation.

Methods

The core of MLModeler5's approach relies on two powerful pre-trained transformer models: RoBERTa and ALBERT. These models, known for their strong performance in natural language understanding tasks, were fine-tuned for the specific subtasks.

For Subtask 1 (News Genre Categorisation), the models were trained to classify articles into one of three categories: Opinion Piece, Satire Piece, or Piece Aiming at Objective Reporting.

For Subtask 2 (Framing Detection), a multi-label classification approach was employed. The models were tasked with identifying which of 14 pre-defined, domain-independent framing dimensions were present in an article. These dimensions cover a broad spectrum, including economic factors, fairness, legal aspects, policy, crime, security, health, quality of life, cultural identity, and political/external reputation.

Key methodological innovations include:

  • Multi-lingual Handling: Articles from six languages (English, French, Italian, German, Polish, Russian) were translated into English for training and validation, with Spanish, Greek, and Georgian added for testing. This allows for a more comprehensive analysis across linguistic boundaries.
  • Data Augmentation: The nlpaug library was used to augment the training data. This technique helps to increase the size and diversity of the dataset, potentially improving model robustness and generalization. Two augmentation strategies were explored, with varying aug_max parameters.
  • Class Weighting: To address the inherent imbalance in the dataset (some genres or frames appearing more frequently than others), class weights were strategically applied during model training. This ensures that the model pays adequate attention to under-represented classes.
  • Model Architectures: Both RoBERTa and ALBERT were fine-tuned using a sequence classification architecture. For Subtask 1, a softmax activation function was used for the final layer, suitable for multi-class classification. For Subtask 2, a sigmoid activation function was employed, which is appropriate for multi-label classification where each label's probability is independent.

Datasets

The study utilized a multi-lingual dataset provided for SemEval-2023 Task 3. This dataset comprised news and web articles in a plain text format.

  • Training and Validation Data: Articles were provided in six languages: English, French, Italian, German, Polish, and Russian.
  • Testing Data: Articles were provided in an additional three languages: Spanish, Greek, and Georgian, along with the previously mentioned six.

The articles covered a range of topics including COVID-19, climate change, abortion, migration, and the Russo-Ukrainian war. The dataset was curated to include both mainstream and alternative news sources, with a significant portion identified as potentially spreading misinformation. The organizers provided all annotations and labels.

Key Insights and Ideas

  • RoBERTa Outperforms ALBERT: Across both subtasks and evaluation metrics, the RoBERTa model consistently achieved better performance than the ALBERT model. This suggests that RoBERTa's architecture and pre-training were more effective for this specific task.
  • The Importance of Class Weights: Adjusting class weights, particularly after initial evaluation, led to significant improvements in F1 scores for both subtasks. This highlights the critical role of addressing data imbalance in achieving optimal performance.
  • Impact of Data Augmentation: The use of nlpaug for text augmentation demonstrated a positive impact on model performance, especially in Subtask 2. This underscores the value of data augmentation techniques in enhancing the robustness of NLP models.
  • Dataset with Numbers is Better: For Subtask 1, the dataset that included numbers generally yielded better results than the dataset without numbers, for both RoBERTa and ALBERT. This suggests that numerical information might be a valuable signal for genre classification.

Key Results

  • Subtask 1 (News Genre Categorisation):
    • The best F1 Macro score achieved was 0.66077 using the RoBERTa model with adjusted class weights (class weights 2) on the dataset that included numbers.
  • The official submission using RoBERTa achieved an F1 Macro score of 0.61632 and an F1 Micro score of 0.62963.
  • The ALBERT model, with adjusted class weights, achieved an F1 Macro score of 0.61503 on the dataset including numbers.
  • Subtask 2 (Framing Detection):
    • The best F1 Micro score achieved was 0.47692 using the RoBERTa model with Text Augmentation 1 parameters during the SemEval evaluation.
  • The official submission using RoBERTa and Text Augmentation 1 achieved an F1 Micro score of 0.47692 and an F1 Macro score of 0.42724.
  • The ALBERT model achieved an F1 Micro score of 0.35455 with Text Augmentation 1 parameters.

Limitations and Future Work

The authors acknowledge several areas for future improvement:

  • Data Pre-processing: Further exploration of data pre-processing techniques, including more sophisticated NLP augmentation methods, could lead to better results.
  • NLP Augmentation Fine-tuning: Fine-tuning the parameters of the NLP augmentation method, such as the number of words substituted or sentences formed, is suggested for enhanced performance.
  • Hyperparameter Tuning: Fine-tuning the hyperparameters of the RoBERTa and ALBERT models is identified as a promising avenue for achieving more accurate results.
  • Multi-lingual Model Training: While translation was used, training models directly on multi-lingual data or exploring cross-lingual transfer learning more deeply could be beneficial.

Summary updated

BibTeX
@inproceedings{khanchandani2023mlmodeler,
  author = {Arjun Khanchandani and Nitansh Jain and Jatin Bedi},
  title = {{MLModeler5 at SemEval-2023 Task 3: Detecting the Category and the Framing Techniques in Online News in a Multi-lingual Setup}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure