HHU at SemEval-2023 Task 3: An Adapter-based Approach for News Genre Classification

Fabian Billert, Stefan ConradACL 2023

TL;DR

This paper tackles the challenge of classifying news articles into "Reporting," "Opinion," or "Satire" across multiple languages, even with limited training data. They achieve strong results by using "AdapterFusion," a technique that intelligently combines specialized language models (adapters) to leverage knowledge from different languages, significantly improving performance, especially in low-resource scenarios and zero-shot predictions.

What Problem Does This Paper Address?

The SemEval-2023 Task 3 aimed to understand what makes news articles persuasive by classifying them into three genres: Reporting, Opinion Piece, or Satire. This is a multilingual task, meaning models need to handle articles in various languages. A key challenge is the limited availability and imbalance of training data for each language, with "Opinion" being over-represented and "Satire" being a minority class. Furthermore, news articles often exceed the typical 512-token limit of transformer models, requiring careful preprocessing. The task also includes a zero-shot setting, where models must perform well on languages they haven't seen during training.

Methods

The authors employ an adapter-based approach, building upon the XLM-RoBERTa multilingual model. Adapters are small, modular neural network components that can be "plugged into" a pre-trained model to adapt it for specific tasks or languages without retraining the entire model.

  1. Task-Adapters: For each language, a task-specific adapter is trained. This adapter learns to perform the genre classification for that particular language.
  2. Language-Adapters: To facilitate cross-lingual transfer, language-specific adapters are also used. These capture language-specific nuances.
  3. AdapterFusion: This is the core novel contribution. Instead of training a single model for each language, AdapterFusion combines multiple task-specific adapters in parallel. A "fusion layer" then learns to dynamically weigh the contributions of each adapter based on the input, effectively creating a more robust and generalized model. This is particularly beneficial for low-resource languages, as it can draw knowledge from better-resourced languages.
  4. Zero-Shot Transfer: For languages with no training data, the authors leverage the trained adapters. They replace the language-adapter of a trained model with the adapter of the target zero-shot language and then use the AdapterFusion model for prediction.

Data augmentation involved translating training data between languages and splitting long articles into smaller subtexts.

Datasets

The primary dataset for this task is the SemEval-2023 Task 3 dataset, which contains multilingual news articles across six languages: English, Italian, German, French, Polish, and Russian. The authors focused their submissions on English, Italian, German, and Russian, but also evaluated on zero-shot languages like Spanish and Greek. They also utilized the French and Polish datasets to augment their training data.

Key Insights and Ideas

  • AdapterFusion is a powerful tool for multilingual low-resource scenarios: By combining specialized adapters, the model can effectively pool knowledge across languages, leading to significant performance gains, especially when individual language datasets are small.
  • Intelligent weighting is key: The AdapterFusion layer's ability to learn how to weigh different task-adapters based on the input is crucial for its success. This dynamic weighting allows it to adapt to the specific needs of each input article.
  • Zero-shot performance can be significantly boosted: AdapterFusion, when used for zero-shot transfer, shows a marked improvement over simply using a single task-adapter, demonstrating its effectiveness in generalizing to unseen languages.
  • Evaluation vs. Test Set Discrepancies: The paper highlights a notable difference in F1-macro scores between the evaluation and test sets for most languages, suggesting potential differences in data distribution or composition that warrant further investigation.

Key Results

  • English: Achieved 4th place out of 27 teams with an F1-macro score of 0.5942 on the test set using AdapterFusion.
  • Greek (Zero-Shot): Achieved 3rd place out of 15 teams with an F1-macro score of 0.7504.
  • General Performance: AdapterFusion consistently outperformed single task-adapters in most languages, particularly in English and Russian.
  • Zero-Shot Transfer: AdapterFusion showed improved performance for zero-shot languages like Greek compared to using only a single task-adapter. For instance, when transferring from Italian to Greek, AdapterFusion achieved a score of 0.7502, while the task-adapter achieved 0.6807.

Limitations and Future Work

  • Language-Specific AdapterFusion: The authors trained separate AdapterFusion models for each language. They suggest that a future direction could be to train a single AdapterFusion model that can handle multilingual data simultaneously, which would be more efficient.
  • Limited Language Coverage: Due to the unavailability of language-adapters for French and Polish within the XLM-R architecture on the AdapterHub, these languages were not directly included in their primary adapter training. Including them could further improve results.
  • Data Imbalance: While they used weighted cross-entropy loss, the inherent data imbalance (especially the minority "Satire" class) remains a challenge.
  • Evaluation vs. Test Set: The significant score differences between evaluation and test sets suggest that future work could explore methods to better align these distributions or investigate the test set labels if possible.

Summary updated

BibTeX
@inproceedings{billert2023hhu,
  author = {Fabian Billert and Stefan Conrad},
  title = {{HHU at SemEval-2023 Task 3: An Adapter-based Approach for News Genre Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure