TL;DR
This paper presents UMUTeam's approach to SemEval-2023 Task 3, a multilingual challenge focused on analyzing online news. They achieved strong results by fine-tuning a multilingual transformer model (XLM-ROBERTa) and using sentence transformers to extract relevant text segments, demonstrating the power of transfer learning for cross-lingual news analysis.
What Problem Does This Paper Address?
In today's digital age, vast amounts of news are published daily across numerous languages and countries. For organizations and researchers, monitoring and understanding this information deluge is crucial for identifying trends, patterns, and how specific topics are presented globally. However, manually processing this multilingual content is time-consuming and inefficient. SemEval-2023 Task 3 aimed to automate this process by identifying key aspects of news articles: their genre (opinion, reporting, satire), the frames used to present information, and the persuasion techniques employed. This task is particularly challenging due to the multilingual nature of the data and the need for nuanced understanding of news content.
Methods
The UMUTeam's approach for Subtasks 1 (News Genre Categorization) and 2 (Framing Detection) involved a multi-stage process:
- Data Merging and Preprocessing: All available training and development data across the six languages were combined. Preprocessing involved cleaning the text by removing social media elements like hyperlinks, hashtags, and mentions, as well as the "*" character and email links.
- Text Chunking with Sentence Transformers: Recognizing that news articles can be very long, making direct processing difficult, the team employed a Sentence-Transformers model (
sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2). This model was used to extract the most relevant part of the news article by identifying sentences or sentence pairs with high similarity to the article's title. The text was divided into 200-token fragments with 50-token overlaps, and the fragment most semantically related to the title was selected. - Fine-tuning XLM-ROBERTa: The core of their classification model was XLM-ROBERTa, a powerful multilingual transformer model pre-trained on a massive dataset across 100 languages. This pre-trained model was then fine-tuned on the combined news dataset for the specific classification tasks. For Subtask 1 (single-label classification), a standard sequence classification layer was added. For Subtask 2 (multi-label classification), a multi-label sequence classification layer was used.
- Hyperparameter Optimization: To find the optimal settings for the XLM-ROBERTa model, the researchers used RayTune with a Tree of Parzen Estimators (TPE) approach. They explored ranges for weight decay, training batch size, number of epochs, and learning rate, ultimately settling on 15 epochs, 0.01 weight decay, a batch size of 8, and a learning rate of 1e-5.
Datasets
The study utilized the dataset provided by the SemEval-2023 Task 3 organizers. This dataset comprised news articles in six languages: English, French, German, Italian, Polish, and Russian. The articles were collected between 2020 and mid-2022 and covered a range of widely discussed topics, including COVID-19, climate change, abortion, migration, and geopolitical events. The dataset was divided into training, development, and testing splits. The authors noted that the training and development sets were relatively small, leading them to combine them for fine-tuning.
Key Insights and Ideas
- Leveraging Multilingual Pre-training: The success of XLM-ROBERTa highlights the significant advantage of using large, multilingual pre-trained models. These models capture rich linguistic knowledge that can be effectively transferred to downstream tasks, even in languages with limited training data.
- The Importance of Text Salience: Directly processing long news articles can be computationally intensive and may dilute the signal from the most important parts. Using sentence transformers to extract title-relevant text segments proved to be an effective strategy for focusing the model's attention.
- Zero-Shot Learning Capabilities: The model demonstrated impressive performance in languages for which no explicit training data was provided (e.g., Spanish, Greek, Georgian), underscoring the power of cross-lingual transfer learning. This suggests that the model can generalize well to unseen languages with similar linguistic structures.
- Dataset Imbalance Challenges: The English news category in Subtask 1 presented a significant challenge due to an unbalanced dataset, particularly with very few examples of satire. This imbalance directly impacted the model's performance on this specific language and category.
Key Results
- Subtask 1 (Genre Categorization): The UMUTeam's system achieved excellent results, outperforming baselines in most languages. Notably, they secured first place in the official rankings for French and German. While performance on English was limited (Macro F1 of 0.29408 on the validation set) due to dataset imbalance, the system showed strong zero-shot performance for languages like Spanish and Greek.
- Subtask 2 (Framing Detection): The system also performed very well in this multi-label task, consistently ranking within the top 10 across all languages. The micro F1-score on the validation set was particularly strong for English (0.73514) and Polish (0.65342). Similar to Subtask 1, the model demonstrated good zero-shot capabilities for languages not included in the training set.
Limitations and Future Work
The authors acknowledge that the dataset imbalance for certain categories (like satire in English news) significantly impacted performance. They also note that the size of the training and development datasets was relatively small, which motivated their decision to combine them.
For future work, the team plans to:
- Explore other multilingual models.
- Investigate fine-tuning a Sentence Transformer model using the news dataset to potentially improve performance.
- Conduct an ablation study on fragment size to better understand its impact.
- Extend their analysis to Spanish splits of evaluation datasets for financial news, hate-speech detection, and political ideology.
Summary updated