TL;DR
This paper tackles the challenge of classifying online news across multiple languages and tasks, particularly focusing on imbalanced datasets. The team QUST achieved strong results by employing a pre-trained multilingual model (XLM-R) and fine-tuning it with strategies like class weights, sample weights, and a novel task-dependent approach, ultimately securing second place in Italian and Spanish for one subtask.
What Problem Does This Paper Address?
The SemEval-2023 Task 3 presents a significant challenge: automatically detecting news genre, framing, and persuasion techniques across six languages, with three additional "surprise" languages for zero-shot evaluation. A major hurdle is the inherent data imbalance within the datasets, especially in Subtask 1, where minority classes like "satire" have very few examples compared to majority classes like "opinion." This imbalance can lead to models overfitting on common classes and failing to learn from rare but important categories. Furthermore, training deep learning models from scratch with limited data is often insufficient, necessitating the use of pre-trained models and effective fine-tuning strategies.
Methods
The QUST team explored both monolingual and multilingual approaches. Initially, they experimented with traditional machine learning and deep learning models in a monolingual setting, employing under-sampling of majority classes to address data imbalance. The core of their strategy, however, lies in fine-tuning a pre-trained multilingual model, specifically XLM-ROBERTa (XLM-R), which can process all the task's languages.
Key techniques include:
- Class Weights: Adjusting the cross-entropy loss to penalize misclassifications of minority classes more heavily.
- Sample Weights: Assigning weights to individual samples to ensure a more balanced representation of the dataset within training batches.
- Task-Dependent Fine-tuning: A novel approach where the model is sequentially fine-tuned across subtasks (from Subtask 1 to Subtask 3). This leverages the idea that while labels differ, the underlying content of the news articles is similar, allowing knowledge transfer between tasks. This contrasts with the standard Task-Agnostic approach, where each subtask is fine-tuned independently.
- Ensembling: The final predictions were made by ensembling the top three models from a 10-fold cross-validation.
Datasets
The experiments were conducted on the SemEval-2023 Task 3 dataset, which includes news articles in English, French, German, Italian, Polish, and Russian, with Spanish, Greek, and Georgian for zero-shot evaluation. The dataset is structured into three subtasks:
- Subtask 1 (T1): Document-level multiclass classification (news genre).
- Subtask 2 (T2): Document-level multilabel classification (framing).
- Subtask 3 (T3): Sentence-level multilabel classification (persuasion techniques).
The paper mentions using v2 and v3 of the released data, with v3 including development set labels to expand the training data.
Key Insights and Ideas
- Multilingual Models Outperform: The study consistently shows that multilingual approaches, particularly XLM-R, are superior to monolingual ones, highlighting the benefit of shared cross-lingual information.
- Data Imbalance is Critical: The effectiveness of class weights and sample weights in mitigating the impact of imbalanced data is clearly demonstrated, leading to significant performance improvements.
- Task-Dependent Fine-tuning is Beneficial: The sequential, task-dependent fine-tuning strategy proved more effective than the task-agnostic approach, suggesting that learning from previous subtasks enhances performance on subsequent ones. This indicates that the model effectively learns shared information across tasks.
- Minority Class Focus: The ablation studies reveal that removing class weights drastically degrades performance, particularly on the minority "satire" class, emphasizing the importance of explicitly addressing data imbalance.
Key Results
- The QUST system achieved the second-best rank in Italian and Spanish (zero-shot) for Subtask 1.
- The proposed method (XLM-R with class weights, sample weights, and task-dependent fine-tuning) significantly improved performance across all subtasks compared to baseline models.
- An ablation study showed that removing class weights led to a 14% drop in average macro-f1. Removing sample weights also negatively impacted performance, especially for minority classes.
- The task-dependent fine-tuning strategy generally outperformed the task-agnostic method.
Limitations and Future Work
- Subtask 3 Performance: The paper notes that their submissions performed significantly worse on Subtask 3 (sentence-level) compared to Subtasks 1 and 2 (document-level). They hypothesize this could be due to the sentence-level nature of T3, which might benefit less from document-level task-dependent fine-tuning, and that early-stage evaluations might not fully reflect T3 performance.
- Future Work: The authors suggest that in future work, back-translating minority class samples from other languages could further augment the data size and improve performance.
Summary updated