A Study on Scaling Up Multilingual News Framing Analysis

Syeda Sabrina Akter, Antonios AnastasopoulosNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of analyzing news framing across multiple languages, which is crucial for understanding global public opinion but hindered by a lack of multilingual datasets. The authors introduce a new crowdsourced dataset and demonstrate that combining it with existing resources significantly improves framing analysis performance, even outperforming expert-annotated data alone. They also explore the potential and limitations of large language models (LLMs) for this task, finding that fine-tuning remains superior to general-purpose LLMs.

What Problem Does This Paper Address?

News framing, the process by which media selectively presents information to influence public perception, is a critical area of study. However, most research and available datasets are limited to English, leaving a significant gap in understanding how framing operates in diverse linguistic and cultural contexts. This limitation makes it difficult to analyze global media narratives and public opinion on international issues. The paper aims to bridge this gap by exploring methods for scaling up multilingual framing analysis, focusing on dataset creation and leveraging automated techniques.

Methods

The core of the paper's methodology involves several key steps:

  • Crowdsourcing for Dataset Creation: The authors utilized a crowdsourcing approach, engaging graduate students in a natural language processing class to annotate news sentences. This method aimed to create a large, albeit potentially "noisy," dataset efficiently.
  • Automatic Translation for Multilinguality: To extend framing analysis to multiple languages, the authors employed automatic translation (specifically Google Translate) to translate existing English datasets (MFC) into 12 typologically diverse languages.
  • Hybrid Dataset Combination: They investigated combining their crowdsourced dataset (SNFC) with the expert-annotated Media Frames Corpus (MFC) to leverage the strengths of both high-quality and high-volume data. They also explored a filtered version of their crowdsourced data (MaSNFC) to mitigate noise.
  • Multilingual Model Training: The study used multilingual transformer models, specifically XLM-RoBERTa, to train and evaluate framing analysis systems on the combined and translated datasets.
  • Large Language Model (LLM) Exploration: The paper also explored the zero-shot performance of prominent LLMs (Mistral, LLaMA-2, Falcon) on the framing analysis task, comparing them to fine-tuned transformer models.

Datasets

  • Media Frames Corpus (MFC): A well-established, expert-annotated dataset for framing analysis, primarily focusing on US news articles across several political issues.
  • Student-Sourced Noisy Frames Corpus (SNFC): A newly introduced dataset created through crowdsourcing by graduate students, covering immigration and same-sex marriage domains.
  • Novel Bengali and Portuguese Benchmarks: The authors created small, novel test sets in Bengali and Portuguese, sourced from local news outlets, to provide a more culturally relevant evaluation for these languages.
  • Translated Datasets: The MFC and SNFC datasets were translated into 12 languages to facilitate multilingual experimentation.

Key Insights and Ideas

  • Crowdsourcing is Viable: The study demonstrates that crowdsourced data, even if "noisy," can significantly boost framing analysis performance when combined with expert-annotated data, offering a cost-effective way to scale up.
  • Quality vs. Quantity Trade-off: While expert annotations (MFC) provide high quality, combining them with a larger volume of crowdsourced data (SNFC) leads to better overall results, suggesting a balance is key.
  • Filtering Improves Crowdsourced Data: Filtering the crowdsourced data to select instances more similar to expert-annotated data (MaSNFC) further enhances performance, indicating that targeted noise reduction is beneficial.
  • LLMs Struggle Without Fine-Tuning: General-purpose LLMs, even large ones, perform poorly on framing analysis in a zero-shot setting compared to task-specific fine-tuned models like RoBERTa. This highlights the importance of specialized training for complex NLP tasks.
  • Cultural Context Matters: The performance drop in multilingual settings, especially for non-Western languages, suggests that automatic translations may not capture cultural nuances, and original news articles from diverse cultural contexts are needed for robust multilingual analysis.

Key Results

  • English Performance: Combining MFC with the crowdsourced SNFC dataset improved accuracy by 2.5 percentage points over MFC alone (72.57% vs. 61.93% for BERT). The best performance was achieved with MFC+MaSNFC, reaching 73.22% accuracy.
  • Multilingual Performance: Accuracies on multilingual test sets were significantly lower than English. For the Bengali test set, MFC10+SNFC achieved the highest accuracy (32.02%). For the Portuguese test set, MFC and MFC+MaSNFC both reached 33.61%.
  • LLM Performance: The best-performing LLM (Mistral-7B-Instruct-v0.1) achieved only 35.33% exact match accuracy on the English MFC test set, significantly lower than the fine-tuned RoBERTa model (73.22%).

Limitations and Future Work

  • Reliance on Automatic Translation: The study acknowledges that using Google Translate for multilinguality may not capture the full nuances of lower-resource languages and can introduce errors. Future work should focus on obtaining original news articles in diverse languages and cultural contexts.
  • US-Centric Bias: The MFC dataset's focus on US news may lead to a bias in the training data, which automatic translations might not fully overcome. The new Bengali and Portuguese test sets attempt to mitigate this, but a broader range of culturally diverse data is needed.
  • LLM Fine-Tuning: The poor performance of LLMs in a zero-shot setting suggests a significant avenue for future research: task-specific fine-tuning of LLMs for framing analysis.
  • Data Imbalance: Error analysis revealed that data imbalance in certain frames (e.g., 'Political', 'Legality, Constitutionality, Jurisdiction') can lead to model bias. Further investigation into addressing this imbalance is warranted.

Summary updated

BibTeX
@inproceedings{akter2024study,
  author = {Syeda Sabrina Akter and Antonios Anastasopoulos},
  title = {{A Study on Scaling Up Multilingual News Framing Analysis}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.260},
}
Figure