Team JUSTR00 at SemEval-2023 Task 3: Transformers for News Articles Classification

Ahmed Al-Qarqaz, Malak AbdullahACL 2023

TL;DR

This paper explores the effectiveness of transformer-based language models for classifying news articles across multiple languages and styles. The team found that models capable of processing longer text sequences, like Longformer, significantly outperform those with shorter context windows, highlighting the importance of context length in this task.

What Problem Does This Paper Address?

The SemEval-2023 Task 3 competition presents a challenging multi-lingual news article classification task. The goal is to build machine learning systems that can categorize news articles based on their nature (e.g., opinion, report, satire) and style (e.g., framing techniques). This task is particularly difficult due to class imbalance within datasets and the inherent limitation of transformer models to process only a fixed, often short, input sequence length (typically 512 tokens). Many news articles far exceed this limit, forcing truncation and potential loss of crucial information.

Methods

The researchers experimented with several state-of-the-art transformer-based language models, including BERT, RoBERTa, BigBird, and Longformer. For classification, they added a fully connected linear layer on top of the pooled output of each transformer. A Softmax activation was used for the multi-class classification subtask (news genre), and a Sigmoid activation for the multi-label classification subtask (news framing). All models were trained using the Adam optimizer with weight decay and a dropout of 0.1. They specifically focused on models that could handle longer input sequences, such as BigBird and Longformer, which employ modified attention mechanisms to overcome the quadratic complexity of standard self-attention.

Datasets

The SemEval-2023 Task 3 competition provided datasets for six languages: English, German, Italian, French, Polish, and Russian. The task was divided into three subtasks:

  • Subtask-1: News Genre Categorization (multi-class classification: opinion, report, satire).
  • Subtask-2: News Framing (multi-label classification of 15 frames).
  • Subtask-3: Persuasion Techniques Detection (multi-label classification of 23 techniques, operating at the paragraph level).

This paper focuses on Subtasks 1 and 2, primarily using the English, German, Italian, French, and Polish datasets. The authors note significant class imbalance within these datasets, which poses a challenge for model training.

Key Insights and Ideas

  • Context Length is King: The most significant finding is that models capable of processing longer sequences (like Longformer and BigBird) outperform those with shorter context windows (BERT, RoBERTa) on this news classification task. This directly addresses the limitation of fixed input lengths in standard transformers.
  • Class Imbalance is a Major Hurdle: The datasets exhibit substantial class imbalance, making it difficult for models to learn to predict less frequent categories. This suggests that future work might need to incorporate strategies to mitigate this imbalance.
  • Preprocessing Potential: While longer context models showed promise, the authors speculate that a preprocessing pipeline, rather than naive training, might yield even better results. This could involve techniques to handle class imbalance or focus on fine-grained analysis of smaller text segments.
  • Multilingual vs. Monolingual Trade-offs: While multilingual models (mBERT, XLM-RoBERTa) were tested, the paper's strongest results came from monolingual models fine-tuned on English data, particularly Longformer. This suggests that for specific language tasks, dedicated monolingual models might be more effective.

Key Results

The Longformer model, with its ability to handle longer sequences (1400 tokens in their experiments), achieved the best performance among the models tested. On the English dataset:

  • Longformer achieved an F1-Micro score of 0.445 on Subtask-1 and an F1-Macro score of 0.305 on Subtask-2 (validation data).
  • On the test data, Longformer achieved an F1-Micro of 0.370 and F1-Macro of 0.256 for Subtask-1 and Subtask-2 respectively.

While not explicitly stated for the test set, the validation results in Table 1 show Longformer outperforming BERT, RoBERTa, and BigBird on both subtasks for English. Table 2 shows the performance of multilingual models on individual languages, with XLM-RoBERTa generally outperforming mBERT.

Limitations and Future Work

  • Computational Resources: The researchers acknowledge that training models with larger input lengths requires significant computational resources, which limits the size of models they can experiment with (e.g., using "base" versions of BERT instead of "large").
  • Preprocessing Neglect: The paper suggests that a preprocessing pipeline could improve results, implying that their current approach of direct training on truncated or long sequences might not be optimal.
  • Limited Multilingual Context Models: Experiments with long-context models were primarily conducted on English data due to the availability of models. The authors note the recent release of BLOOM, a multilingual model with a larger context size, as a promising avenue for future research.
  • Fine-grained Analysis: The authors hint that focusing on fine-grained analysis of small text segments, rather than processing entire articles, might be a more effective strategy, especially given the nature of some labels in the task.

Summary updated

BibTeX
@inproceedings{alqarqaz2023team,
  author = {Ahmed Al-Qarqaz and Malak Abdullah},
  title = {{Team JUSTR00 at SemEval-2023 Task 3: Transformers for News Articles Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure