Media Bias, the Social Sciences, and NLP: Automating Frame Analyses to Identify Bias by Word Choice and Labeling

Felix HamborgACL 2020

TL;DR

This paper introduces a novel approach to automatically detect "bias by word choice and labeling" (WCL) in news articles, a subtle but powerful form of media bias. By combining social science theories of media framing with advanced NLP techniques like cross-document coreference resolution and target-dependent sentiment classification, the research aims to equip news readers with tools to better understand media framing.

What Problem Does This Paper Address?

Media bias is a pervasive issue that can significantly influence public opinion and political outcomes. One particularly insidious form of bias is "bias by word choice and labeling" (WCL). This occurs when journalists use different terms to describe the same concept, subtly framing it in a way that elicits different reader perceptions. For instance, referring to a group as "freedom fighters" versus "terrorists" evokes vastly different emotional and cognitive responses. While social scientists have developed sophisticated methods for analyzing WCL bias, these are often manual, time-consuming, and don't scale to the vast amount of news content produced daily. Conversely, existing computational approaches often lack the nuanced understanding of bias developed in the social sciences, leading to superficial or inconclusive results. This paper bridges this gap by proposing an automated system that leverages both disciplinary insights to identify WCL bias.

Methods

The proposed methodology integrates two core NLP tasks within a broader analysis pipeline:

  • Broad Cross-Document Coreference Resolution (CDCR): This is a crucial component because WCL bias often relies on journalists using seemingly unrelated or even contradictory terms to refer to the same entity or concept across different articles. Standard CDCR struggles with these "broader" coreferences. The approach uses a sieve-based method that analyzes specific characteristics of candidate phrases to determine if they refer to the same semantic concept, even when the wording is significantly different (e.g., "coalition forces" vs. "invading forces").
  • Frame Identification (using Target-Dependent Sentiment Classification - TSC): Once coreferences are resolved, the system aims to understand how a concept is framed. This involves estimating the perception of a semantic concept, moving beyond simple positive/negative sentiment. The approach plans to devise neural models that consider news-specific characteristics, such as the influence of global context on sentiment, and importantly, identify "sentiment shifts" and potential causes and effects of framing. This goes beyond traditional TSC by looking at how the language used sways perception.

The overall pipeline involves preprocessing, broad CDCR, frame identification, and finally, visualization of the results for non-expert users.

Datasets

  • NewsWCL50: A newly created test dataset for evaluating broad CDCR in the context of WCL bias. It consists of 50 news articles manually annotated for coreferential mentions of persons, actions, and abstract concepts.
  • AllSides, NewsWCL50, and POLUSA: Datasets used for research and evaluation of the system's visualization capabilities. These datasets are noted for their diversity in political slant.
  • ECB+ and NIdent: Datasets commonly used for evaluating CDCR approaches, which will be used for future evaluations.
  • Preliminary TSC dataset: A dataset of 3000 sentences with target mentions and sentiment labels, created analogously to established TSC datasets, used for training and testing the frame identification component.

Key Insights and Ideas

  • Interdisciplinary Synergy: The core idea is that combining social science theories of media framing with advanced NLP is essential for tackling complex phenomena like WCL bias, which often operate "between the lines" of text.
  • Beyond Standard Coreference: Recognizing that WCL bias exploits the flexibility of language, the research explicitly addresses the challenge of resolving coreferences that go beyond simple synonyms or pronouns, including contradictory terms.
  • Framing as Perception Shift: The paper frames WCL bias not just as sentiment, but as a shift in how a concept is perceived, influenced by the specific words and phrases used. This necessitates a more nuanced approach than traditional sentiment analysis.
  • User-Centric Visualization: The ultimate goal is to empower news readers. The system's output will be visualized in an accessible way, similar to popular news aggregators, to help users understand the subtle biases present in news coverage.

Key Results

  • CDCR Performance: Preliminary evaluations on the NewsWCL50 dataset show the proposed CDCR approach achieves an F1 score of 45.7 (or 84.4 when evaluated on technically feasible annotations), significantly outperforming a baseline that achieved 29.8 (or 42.1).
  • TSC Challenges: The paper notes that standard TSC methods perform poorly on news texts due to their subtle connotations and lack of news-tailored resources. Average recall for TSC on news articles is around 70.0, lower than on other domains like Twitter or restaurant reviews.

Limitations and Future Work

  • CDCR Limitations: The current CDCR approach primarily focuses on noun phrases (NPs) and excludes verb phrases (VPs). Future work will extend this to include activities (VPs). The performance of the current CDCR approach is still considered preliminary, and further comparison with state-of-the-art methods is planned.
  • TSC Limitations: The paper acknowledges the difficulties in accurately performing TSC on news texts due to subtle language and the need for domain-specific resources.
  • Future Work:
    • Improve CDCR performance by devising methods that leverage language models like BERT.
  • Develop and evaluate the frame identification neural models.
  • Conduct a user study to evaluate the effectiveness of the visualization system.
  • Create a second, larger dataset for CDCR evaluation with improved annotation guidelines.
  • Extend the system to identify other forms of media bias (e.g., commission, omission, picture selection).
  • Integrate the developed methods into popular news aggregators.

Summary updated

BibTeX
@inproceedings{hamborg2020media,
  author = {Felix Hamborg},
  title = {{Media Bias, the Social Sciences, and NLP: Automating Frame Analyses to Identify Bias by Word Choice and Labeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure