TL;DR
This paper tackles the challenge of identifying how named entities are "framed" in multilingual news articles, a crucial step in combating disinformation. The authors explore data augmentation, external knowledge, and class weighting with BERT models, finding that data augmentation was the most effective technique for improving performance.
What Problem Does This Paper Address?
The proliferation of online news has made it increasingly difficult to discern factual information from disinformation and propaganda. Identifying "entity framing" – understanding the roles assigned to named entities within news narratives – is a key component in analyzing and combating manipulative content. SemEval 2025 Task 10 specifically addresses this by asking participants to assign roles to named entities in news articles across five languages (Bulgarian, English, Hindi, Portuguese, and Russian). This task is particularly challenging due to class imbalance, where some entity roles are much rarer than others, making it difficult for standard machine learning models to learn effectively.
Methods
The researchers experimented with several approaches to improve BERT-based models for entity framing:
- Data Augmentation: They employed back-translation and pivot translation to generate synthetic training data, particularly for underrepresented minority classes. This aims to increase the diversity of the training set and expose the model to more examples of rare roles.
- External Knowledge Integration: The team explored incorporating knowledge from sources like Wikipedia/Wikidata and ConceptNet. The idea is to enrich the model's understanding of entities and their relationships, potentially leading to more accurate role assignments.
- Class Weighting: To address the class imbalance issue, they implemented two class weighting strategies:
- Scikit-learn's
compute_class_weight: This assigns weights inversely proportional to class frequencies. - Logarithmic Weighting: This method smooths extreme weight differences by using a logarithmic transformation, aiming for a more gradual adjustment than traditional frequency-based weighting.
- Model Architectures: They fine-tuned BERT-family models, specifically RoBERTa and DistilBERT, using Hugging Face's API.
Datasets
The primary dataset used was from SemEval 2025 Task 10, which includes news and web articles from two domains: the Ukraine-Russia War and Climate Change. The authors initially worked with an English dataset of 328 articles. For multilingual experiments, they utilized a larger dataset, ultimately training on 12,450 multilingual samples with a 20% validation split. The paper also mentions using a development set without annotations for training and evaluation, and a test set for final submission.
Key Insights and Ideas
- Data Augmentation is King (for this task): The study found that data augmentation, particularly back-translation and pivot translation, was the most effective technique for improving classification performance, outperforming external knowledge integration and class weighting.
- The Perils of Overfitting: The researchers observed that their development set results were better than their final submission results, suggesting that their models might have been overfitting to the development data, leading to overly optimistic performance estimates.
- Class Imbalance is a Major Hurdle: The paper explicitly highlights class imbalance as a significant challenge, noting that standard machine learning models struggle with minority classes. Their exploration of class weighting methods was a direct response to this.
- Dynamic Thresholding Matters: Adjusting the classification threshold from 30% to 20% significantly improved the Exact Match Ratio for DistilBERT, underscoring the importance of fine-tuning this parameter.
Key Results
The paper reports that their model ranked 21st out of 32 participants in the SemEval 2025 Task 10 submission. While their development set performance was stronger, indicating potential overfitting, the study did show improvements over a baseline.
- Data Augmentation: This approach showed the most significant positive impact on performance.
- RoBERTa vs. DistilBERT: RoBERTa generally outperformed DistilBERT in initial training evaluations, showing more consistent performance.
- External Knowledge: While external knowledge integration showed some promise, it did not yield as substantial improvements as data augmentation. The paper notes that while it achieved high main role accuracy, other metrics were lower, suggesting potential overfitting or issues with dataset distribution.
Limitations and Future Work
The authors acknowledge several limitations:
- Overfitting: The discrepancy between development and test set performance strongly suggests overfitting, possibly due to the similarity between training and validation data.
- "Over-Augmentation" Risk: They noted that their best results came from an intermediate level of data augmentation, implying that excessive augmentation could introduce noise or degrade performance.
- Lack of Detailed Stage Recording: A key oversight was not recording submission results at various stages of experimentation, making it difficult to pinpoint the exact impact of each technique.
- Need for Further Investigation: The paper suggests that more refined augmentation techniques are needed to reduce noisy data. They also propose exploring other models, such as GPT-based architectures, and ensemble learning techniques for future work. The potential benefits of prompting techniques were also noted but could not be fully explored due to time constraints.
Summary updated