TL;DR
Standard sentiment analysis usually assumes a single author expressing feelings toward a clear topic, failing in political news where multiple actors blame, criticize, or endorse each other within the same sentence. This paper formalizes the task of directed sentiment extraction (who holds what sentiment toward whom) and introduces DSE2QA, a method that reframes 5-way relation classification into binary question-answering queries over a pretrained RoBERTa backbone. Evaluated on a newly crowdsourced dataset of over 16,000 sentences, DSE2QA outperforms standard classification baselines and reveals systemic framing differences across partisan media.
What Problem Does This Paper Address?
In political reporting, sentences rarely carry a single, monolithic sentiment. A sentence like "X supported Y for criticizing Z" contains multiple conflicting polarities: positive from $X \to Y$, negative from $Y \to Z$, and negative from $X \to Z$.
Existing sentiment analysis pipelines typically focus on document/sentence-level polarity or target-dependent sentiment, implicitly assuming the speaker is the document author. Meanwhile, relation extraction typically targets factual relations (e.g., EMPLOYEE_OF) rather than affective, subjective stances. To bridge this gap, the authors formulate directed sentiment extraction: given a sentence $s$ containing entity mentions $p$ and $q$, determine whether the relation between them is neutral, positive ($p \to q$ or $q \to p$), or negative ($p \to q$ or $q \to p$).
Methods
Instead of using a standard 5-class classification head atop a pooled text representation, the authors propose DSE2QA (Directed Sentiment Extraction to Question-Answering):
- Input Transformation & Masking: Entity mentions $p$ and $q$ in sentence $s$ are replaced with abstract tokens
[ENT1]and[ENT2]. - Auxiliary Question Generation: The problem is recast into five separate yes/no verification tasks. For each candidate class $i$, an auxiliary query $q_i$ is constructed. The authors compare two query formats:
- Complete questions (natural language): e.g., "Does [ENT1] have positive sentiment toward [ENT2]?"
- Pseudo questions (concise keyword format): e.g., "[ENT1] - [ENT2] - positive".
- Inference via Binary Scoring: A pretrained
RoBERTa-basemodel processes[CLS] s [SEP] q_i [SEP], trained with binary cross-entropy. During inference, the model evaluates all five queries and selects the class with the maximum confidence score: $\hat{y} = \arg\max_i y_i$.
Datasets
- Newly Introduced Benchmark: Drawn from the RealNews corpus (filtered to reliable outlets via Media Bias/Fact Check). Sentences containing at least two named entities (persons, groups, organizations, locations) were sampled using a mix of sentiment lexicons and random selection.
- Crowdsourced Annotations: 16,228 sentences annotated via Amazon Mechanical Turk (5 workers per sentence, majority vote; Fleiss’ $\kappa = 0.26$). The resulting distribution is skewed: 10,604 neutral, 1,983 positive, and 3,641 negative pairs.
- Downstream Case Study Corpora:
- 2016 US Election: 39,000 articles sampled across 15 media outlets.
- COVID-19 Pandemic Coverage: 6,180 articles from Common Crawl covering 35 left-, center-, and right-leaning news outlets (over 1 million analyzed entity pairs).
Key Insights and Ideas
- Keyword prompts beat natural language queries: Counterintuitively, minimal "pseudo questions" (keyword triplets) outperformed fully formed syntactic questions (Micro F1 0.7973 vs. 0.7726). A few prompt keywords provide sufficient conditioning for the transformer without introducing syntactic distractors.
- Positive relations are significantly harder to detect: Across all models, negative directed sentiments were detected with much higher F1 scores than positive relations. The authors attribute this to the inherently adversarial and critical nature of political discourse, where positive endorsements are subtler or more context-dependent.
- Media bias emerges in entity targets, not just tone: In the COVID-19 case study, the rank correlation of negatively targeted entities between left-leaning and right-leaning media was negative ($\rho = -0.5722, p < 0.001$). Left-leaning outlets framed COVID-19 as a domestic crisis centered on Donald Trump, whereas right-leaning outlets directed negative coverage outward toward foreign actors (China and Chinese officials).
Key Results
- Test Set Performance: DSE2QA (Pseudo) achieved the best overall performance, outperforming a fine-tuned RoBERTa multi-class classifier and prior BiLSTM-based relation extraction baselines (Liang et al., 2019):
- DSE2QA (Pseudo): 0.7973 Micro F1 | 0.6766 Macro F1 | 0.7488 mAP
- DSE2QA (Complete): 0.7726 Micro F1 | 0.6617 Macro F1 | 0.7387 mAP
- Standard RoBERTa: 0.7486 Micro F1 | 0.6409 Macro F1 | 0.7319 mAP
- LNZ (Combined BiLSTM): 0.7055 Micro F1 | 0.5358 Macro F1 | 0.5295 mAP
- Per-Class Breakdown: Non-transformer baselines struggled heavily on non-neutral directional relations (e.g., LNZ scored only 0.3333 F1 on right-to-left positive relations, where DSE2QA achieved 0.5672).
Limitations and Future Work
- Sentence-Level Constraint: The current task formulation only identifies relations between entities within a single sentence, ignoring cross-sentence discourse, coreference, and document-level sentiment flows.
- Pipeline Dependency: The system assumes entity spans are already identified by an upstream NER pipeline (e.g., spaCy), which can propagate extraction errors. The authors recommend joint NER and directed sentiment extraction for future work.
- Annotator Agreement: A Fleiss' $\kappa$ of 0.26 reflects high subjectivity and ambiguity when interpreting political stances without external context.
Summary updated