Abstract
Understanding who blames or supports whom in news text is a critical research question in computational social science. Traditional methods and datasets for sentiment analysis are, however, not suitable for the domain of political text as they do not consider the direction of sentiments expressed between entities. In this paper, we propose a novel NLP task of identifying directed sentiment relationship between political entities from a given news document, which we call directed sentiment extraction. From a million-scale news corpus, we construct a dataset of news sentences where sentiment relations of political entities are manually annotated. We present a simple but effective approach for utilizing a pretrained transformer, which infers the target class by predicting multiple question-answering tasks and combining the outcomes. We demonstrate the utility of our proposed method for social science research questions by analyzing positive and negative opinions between political entities in two major events: 2016 U.S. presidential election and COVID-19. The newly proposed problem, data, and method will facilitate future studies on interdisciplinary NLP methods and applications.
Introduction
In the realm of political news, sentiment is rarely a global property of a sentence. Instead, it is a complex web of interactions where multiple actors—politicians, countries, and organizations—constantly blame, endorse, or criticize one another. Standard sentiment analysis tools often fail in this context because they assign a single polarity to a text or focus on a specific target without identifying the source of the sentiment.
To address this, we introduce the task of directed sentiment extraction (DSE). This task requires identifying both the source (who holds the opinion) and the target (who the opinion is about) to determine the specific sentiment relationship (e.g., Entity A blames Entity B).
The DSE2QA Framework
Our core methodological contribution is DSE2QA (Directed Sentiment Extraction to Question-Answering). Rather than treating sentiment extraction as a standard 5-class classification problem, we reframe it as a series of binary question-answering (QA) sub-tasks.
As shown in the figure above, the model takes a news sentence and generates auxiliary questions such as "Does Donald Trump have negative sentiment toward China?" We utilize a pretrained RoBERTa model to process these questions alongside the original text. We found that "Pseudo" questions—using simple keyword strings like [ENT1] - [ENT2] - negative—actually outperformed full natural language questions, suggesting that the transformer effectively leverages semantic keywords over grammatical structure.
Data and Results
To train and evaluate our models, we created the Directed Sentiment Dataset, a benchmark of 16,228 manually annotated sentences from the Real News corpus. Our DSE2QA approach achieved a Micro F1 of 0.7973, significantly outperforming standard classification baselines.
Our analysis revealed that negative sentiment is far more prevalent in political news than positive sentiment. In a case study of the 2016 U.S. election, negative relations were nearly five times more common than positive ones. Furthermore, the model proved capable of identifying subtle media biases. For example, during the COVID-19 pandemic, left-leaning media outlets frequently framed Donald Trump as the target of blame for domestic issues, while right-leaning outlets framed him as the source of negative sentiment directed toward China.
Conclusion
By reframing sentiment extraction as a QA task, we provide a robust tool for computational social scientists to quantify political narratives. While currently limited to sentence-level extraction, future work will expand this framework to document-level inference and joint entity recognition, further enhancing our ability to map the complex landscape of media discourse.
Summary updated