Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech

Jonathan Pofcher, Christopher M. Homan, Randall Sell, Ashiqur R. KhudaBukhshEMNLP 2025

TL;DR

This paper introduces a novel dataset and methodology for analyzing user comments on LGBTQ+ news content across major US cable news outlets. It reveals significant political biases in how users and even LLMs interpret this content, highlighting a critical gap in understanding and fostering positive online discourse for marginalized communities.

What Problem Does This Paper Address?

The internet and social media offer both opportunities and perils for the LGBTQ+ community. While they can provide safe spaces and resources, they also expose users to cyberbullying, dehumanization, and homophobic biases in AI. Understanding how users engage with LGBTQ+ news content in mainstream media is crucial for identifying and mitigating harmful discourse. Existing research often focuses on hate speech and uses limited news sources. This paper aims to bridge this gap by analyzing user interactions with LGBTQ+ news across major US cable news outlets, specifically focusing on distinguishing "hope speech" (positive, supportive content) from negative and neutral content.

Methods

The authors employ a multi-stage approach:

  1. Data Collection: They collected over 1.4 million YouTube comments from over 3,100 news videos from Fox News, CNN, and MSNBC.
  2. Video Selection: A two-step process using GPT-4o mini was used to identify videos specifically relevant to the LGBTQ+ community.
  3. Annotation Framework: A collaborative human-LLM annotation framework was used to build a dataset of 3,750 comments. This involved an initial seed set, followed by active learning stages (certainty and margin sampling) to address class imbalance, particularly for minority classes like "neutral" and "negative" comments.
  4. LLM Classification: They fine-tuned Mistral and Llama 3 models on their annotated dataset to classify comments into four categories: positive (hope speech), negative, neutral, and irrelevant. They also explored zero-shot classification and models fine-tuned on politically aligned subsets of the data.
  5. Analysis: The study analyzes user engagement metrics (dislike rates, engagement rates), political biases in human annotations, LLM performance across political affiliations, and performs lexical analysis of comments.

Datasets

  • YouTube Comments Corpus: Over 1.4 million comments from over 3,100 YouTube videos from Fox News, CNN, and MSNBC.
  • Dhope Dataset: A newly curated dataset of 3,750 annotated YouTube comments, categorized as positive (hope speech), negative, neutral, or irrelevant. This dataset was built through a rigorous annotation process involving crowd-sourced annotators with diverse political leanings and self-identified LGBTQ+ individuals.

Key Insights and Ideas

  • Political Bias is Pervasive: Human annotators' political beliefs strongly influence their labeling of LGBTQ+ content, with significant differences observed between Republicans, Democrats, and Independents.
  • LLMs Inherit Bias: Fine-tuned LLMs reflect the political biases present in the training data, with models trained on Republican-annotated data showing distinct patterns from those trained on Democrat or Independent data. Zero-shot LLMs tend to align more with liberal raters.
  • Hope Speech is Scarce: Across all news channels, negative comments significantly outweigh hope speech, indicating a societal tendency towards critical or oppositional engagement with LGBTQ+ topics online.
  • Channel Differences: Fox News exhibits the highest proportion of negative comments and the lowest hope speech, aligning with its reputation for conservative viewpoints. MSNBC and CNN show more balanced or slightly supportive discourse, with MSNBC having the highest "positivity ratio."
  • Lived Experience Matters: LGBTQ+ annotators were more likely to identify relevant content and less likely to label it as irrelevant compared to non-LGBTQ+ annotators, suggesting that lived experience aids in understanding nuanced discourse.

Key Results

  • Annotation Agreement: Moderate inter-rater agreement (Fleiss' κ = 0.431) was achieved, which is in line with extensive prior literature on subjective annotation tasks.
  • LLM Performance: Fine-tuned LLMs (Llama 3 and Mistral) significantly outperformed zero-shot models, with Llama 3 trained on the margin-sampled dataset achieving the best performance.
  • Dislike Ratios: LGBTQ+ videos consistently received higher dislike rates across all three channels compared to non-LGBTQ+ content, with FOX News and CNN showing the most pronounced differences.
  • Positivity Ratio: MSNBC had the highest positivity ratio (32.7%), followed by CNN (30.81%), both above the overall average (22.25%). Fox News had a considerably lower positivity ratio (8.7%).
  • Lexical Analysis: Positive comments often used terms related to rights and freedoms, while negative comments frequently invoked religious frameworks and moral values.

Limitations and Future Work

  • Platform Specificity: The study's focus on YouTube comments limits generalizability to other platforms like Twitter or Reddit, which have different demographics and interaction patterns.
  • Political Simplification: The categorization of political affiliations into Democrat, Republican, and Independent may oversimplify the nuances of political views.
  • Intersectionality: The study did not deeply explore intersectionality with factors like race, ethnicity, or age, which can significantly influence perceptions of LGBTQ+ content.
  • LLM Nuance: LLMs still struggle with sarcasm and subtle forms of bias, requiring further exploration of advanced language models and tuning strategies.
  • Future Work: The authors suggest expanding to other platforms, conducting more in-depth analysis of specific LGBTQ+ communities (e.g., the trans community), exploring participatory AI, and developing more nuanced annotation guidelines.

Summary updated

BibTeX
@inproceedings{pofcher2025hope,
  author = {Jonathan Pofcher and Christopher M. Homan and Randall Sell and Ashiqur R. KhudaBukhsh},
  title = {{Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure