TL;DR
This paper explores news genre and propaganda detection using pre-trained language models, comparing monolingual and multilingual approaches. They found that multilingual models generally outperform monolingual ones, and that fine-tuning on additional relevant corpora and upsampling imbalanced datasets significantly boosts performance, especially for the English language.
What Problem Does This Paper Address?
The paper tackles the challenge of "non-topical text classification," which involves identifying text properties unrelated to its subject matter. Specifically, it focuses on SemEval-2023 Task 3, which has three subtasks: news genre categorization (opinion, reporting, satire), text framing, and persuasion technique detection. A key difficulty in this area is the robustness of classifiers, which can struggle when encountering texts that differ significantly from their training data. This is particularly true for tasks involving multiple languages and potentially imbalanced datasets, where a model might overfit to dominant classes or perform poorly on less represented genres.
Methods
The authors employed several strategies to improve classification performance:
- Model Selection: They experimented with various pre-trained language models, including multilingual BERT (mBERT), XLM-RoBERTa (XLM-R), and their monolingual counterparts for languages like Russian, Polish, German, and French. They also explored larger versions of these models.
- Ensembling: To mitigate individual model weaknesses, they created ensembles by combining the predictions of different models, weighting them based on performance on development sets.
- Additional Data Fine-tuning: A significant aspect of their work was fine-tuning models on external datasets relevant to genre classification, such as the FTD (Functional Text Dimensions) dataset, and also incorporating data from specific sources like "Babylon Bee" (satire) and "Giga News" (reporting).
- Confidence of Prediction: They implemented a confidence metric to identify unreliable predictions. This metric, based on applying dropout during inference and pooling probability distributions, helps in determining how certain a model is about its classification. They explored using this confidence to either reject uncertain predictions or replace them with the majority class.
- Data Augmentation: Upsampling techniques were used to address class imbalance, particularly for the "satire" and "reporting" genres, which were less frequent than "opinion."
Datasets
The primary datasets used were the official SemEval-2023 Task 3 training and development sets for various languages. Additionally, they incorporated:
- FTD (Functional Text Dimensions): An existing dataset for text genre classification in English and Russian.
- Giga News: A dataset for English reporting texts.
- Babylon Bee: An American satirical news website.
- Panorama: A Russian satirical news website.
Key Insights and Ideas
- Multilingual Models Shine: For most languages, multilingual models (mBERT, XLM-R) generally outperformed their monolingual counterparts, highlighting the power of cross-lingual transfer learning.
- Fine-tuning is Crucial: Fine-tuning on additional, relevant corpora (like FTD) significantly improved performance, especially for languages where the original training data was limited. This suggests that leveraging external genre-specific data is highly beneficial.
- Data Balancing Matters: Upsampling underrepresented classes (satire, reporting) was essential for improving performance across languages, indicating that balanced datasets are critical for robust classification.
- Confidence is Key for Robustness: The confidence of prediction metric proved valuable, particularly for the English dataset, which was deemed noisier. Replacing low-confidence predictions with the majority class led to substantial improvements in F1 macro scores.
- Language-Specific Nuances: While multilingual models were generally superior, exceptions like German and Polish showed that monolingual models could sometimes achieve the best results, possibly due to language-specific characteristics (e.g., German compound words).
- Ensembles Offer Reliability: Ensembles, while not always improving raw scores, were found to be more reliable and provide more confident predictions, especially when individual models had differing strengths.
Key Results
- For English, replacing low-confidence predictions with the majority class resulted in a significant growth of the F1 macro score.
- For Russian, the best result was achieved with a mBERT model first fine-tuned on the Russian FTD dataset and then on the Subtask 1 data. Ensembles did not improve Russian results.
- The confidence of prediction metric was found to be more reliable than simply using the maximum probability for Russian.
- For surprise languages (Georgian, Greek, Spanish), a simple multilingual BERT trained on concatenated data from all available languages achieved respectable rankings (3rd for Georgian, 5th for Greek).
- The paper presents detailed F1 macro and micro scores for various model and data combinations across English, French, German, Italian, Polish, and Russian in numerous tables. For instance, on the English dev set (Table 2), some ensemble models with additional data and majority class replacement achieved F1 macro scores around 0.45-0.48.
Limitations and Future Work
- Dataset Size: The authors note that the available training corpora were not sufficient, emphasizing the importance of additional relevant data.
- Language-Specific Biases: They observed that while models performed well on many languages, English showed comparatively lower scores, suggesting potential biases in the English dataset.
- Hyperparameter Optimization: While they used a fixed set of hyperparameters derived from prior work for learning rate and epochs, further fine-tuning could potentially yield better results.
- Surprise Languages: For the surprise languages, the lack of training and development sets limited the extent of experimentation.
- Future Work: The paper suggests that further exploration of ensemble strategies and more sophisticated methods for selecting optimal thresholds for multilabel classification could be beneficial. They also hint at the potential for language-specific tokenization to be crucial for certain languages.
Summary updated