UM6P at SemEval-2023 Task 3: News genre classification based on transformers, graph convolution networks and number of sentences

Hamza Alami, Abdessamad Benlahbib, Abdelkader El Mahdaouy, Ismail BerradaACL 2023

TL;DR

This paper tackles the challenge of automatically classifying news articles into genres (opinion, reporting, satire) using an ensemble of deep learning models. Their approach, combining Longformer, RoBERTa, Graph Convolutional Networks (GCNs), and a simple sentence count model, achieved a respectable 12th place in the SemEval-2023 Task 3 competition.

What Problem Does This Paper Address?

In today's information-saturated world, understanding the nuances of online news is crucial for everything from content curation to political analysis. Automatically categorizing news articles by genre is a fundamental step in this process. The SemEval-2023 Task 3 aimed to advance the state-of-the-art in this area, specifically focusing on English news documents and introducing a dataset designed to test genre, framing, and persuasion detection. This paper addresses the subtask of news genre categorization, aiming to build a robust classifier that can accurately distinguish between different types of news content.

Methods

The authors employ an ensemble strategy, combining the strengths of four distinct models:

  • Longformer: A transformer-based model specifically designed to handle long documents, which is crucial for news articles that can be quite lengthy. It uses a sliding window attention mechanism and a global attention mechanism to capture both local and long-range dependencies.
  • RoBERTa: A robustly optimized BERT approach, another powerful transformer-based language model known for its strong performance on various NLP tasks. For this task, they used XLM-RoBERTa, a multilingual variant, but capped the input sequence length to 512 tokens, taking the first 256 and last 256 tokens to retain potentially important information from the beginning and end of articles.
  • Graph Convolutional Networks (GCNs): This model treats text documents as graphs. Words are nodes, and relationships (like co-occurrence) form the edges. GCNs then learn representations by aggregating information from neighboring nodes, allowing them to capture structural relationships within the text. They used fastText for word embeddings and PyTorch Geometric for GCN implementation.
  • Sentence Number-Based Model: A simpler, yet effective, model that leverages the statistical distribution of sentence counts within each genre. It uses a Gaussian probability function to model this distribution and predict genre based on the number of sentences in a document.

The final prediction is made using a "max voting" mechanism, where the genre predicted by the majority of the individual models is chosen.

Datasets

The primary dataset used is from the SemEval-2023 Task 3.

  • Training Set: Contains 433 English text documents with genre labels (opinion, reporting, satire). The distribution is imbalanced: 88.22% opinion, 9.47% reporting, and 2.31% satire.
  • Validation Set: Consists of 83 documents, with a different distribution: 65.06% reporting, 24.10% opinion, and 10.84% satire.
  • Test Set: Contains 54 documents without labels, used for final evaluation.

To address the imbalanced training data, the authors employed a weighted random sampling technique, ensuring that each batch during training had a balanced representation of samples from different classes.

Key Insights and Ideas

  • Ensemble Power: The core idea is that combining diverse models (transformers, graph-based, statistical) can lead to more robust and accurate predictions than any single model alone.
  • Handling Long Documents: The choice of Longformer highlights the importance of models capable of processing entire news articles, rather than just truncated versions, to capture comprehensive context.
  • Simplicity Can Be Effective: The inclusion of a sentence number-based model, despite its simplicity, demonstrates that even basic statistical features can contribute meaningfully to genre classification, especially when combined with more complex models.
  • Data Imbalance Mitigation: The use of weighted random sampling is a practical and effective strategy for dealing with imbalanced datasets, a common challenge in real-world classification tasks.

Key Results

The ensemble model achieved an F1 macro score of 39.35% on the English test set, ranking 12th in the SemEval-2023 Task 3 leaderboard. Individually, Longformer and RoBERTa performed best among the transformer models, outperforming the GCN and sentence number-based models. The ensemble approach improved the results by approximately 7.4% F1 macro compared to the best individual models.

Limitations and Future Work

  • Limited Scope: The paper focuses solely on English texts for Subtask 1. The broader SemEval task included multiple languages, suggesting potential for multilingual extensions.
  • Data Truncation for RoBERTa: While XLM-RoBERTa was used, the input was truncated to 512 tokens. This might have led to the loss of some information from longer documents.
  • Future Directions: The authors express hope that their approach will inspire further research in document genre classification. Potential future work could involve exploring more sophisticated ensemble techniques, investigating other graph-based methods, or extending the approach to handle other languages and subtasks within the SemEval competition.

Summary updated

BibTeX
@inproceedings{alami2023um,
  author = {Hamza Alami and Abdessamad Benlahbib and Abdelkader El Mahdaouy and Ismail Berrada},
  title = {{UM6P at SemEval-2023 Task 3: News genre classification based on transformers, graph convolution networks and number of sentences}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure