TL;DR
This paper proposes a novel approach to bias detection by using Large Language Models (LLMs) as "readers" to generate comments that capture reader perspectives. This method significantly improves bias detection performance, especially for smaller open-source LLMs, by mimicking how human readers interpret content.
What Problem Does This Paper Address?
Detecting bias in media content is crucial for information integrity and inclusivity. Traditional bias detection methods primarily focus on the writer's perspective, analyzing textual features related to the author's intent. However, this overlooks the reader's perspective, which is equally important as bias perception is influenced by individual interpretations, social norms, and cognitive frameworks. This paper aims to bridge this gap by exploring how LLMs can simulate reader perspectives to enhance bias detection.
Methods
The core innovation lies in framing LLMs as "readers" that generate comments reflecting how a human might perceive a piece of text. The proposed workflow involves three main components:
- Reader (Comment Generation): An LLM (specifically Llama3.1-70B in experiments) is prompted to generate comments that capture diverse viewpoints, emotions, or values evoked by the original content. These prompts are categorized into "General" (external, rational aspects like portrayals and values) and "Individual" (internal, emotional responses like feelings and sharing willingness).
- Selector (Comment Selection): A BERT-based model is trained to distinguish between "positive" (helpful) and "negative" (useless) generated comments. This selector filters comments to ensure only the most beneficial ones are used.
- Detector (Bias Detection): The original text is concatenated with the selected reader-perspective comments, and this combined input is fed into a Detector LLM (e.g., GPT-4, Llama-8B) for bias inference. This approach leverages the reader's interpretation as an additional signal for bias detection.
Datasets
The study evaluates its method on two existing datasets:
- BASIL (news bias): This dataset contains approximately 8,000 sentences labeled as informational bias, lexical bias, or unbiased.
- BeyondGender (gender bias): This dataset comprises over 13,000 English social media posts, labeled for sexism, gender bias, misogyny, and misandry.
Key Insights and Ideas
- LLMs as Readers: The most significant insight is that LLMs can effectively simulate reader perspectives by generating comments, which demonstrably improves bias detection capabilities. This shifts the focus from authorial intent to reader interpretation.
- Emotion Over Value: Emotion-related comments appear to be more beneficial for bias detection than value-related ones, suggesting that capturing the emotional resonance of content is key.
- Small LLMs Benefit Most: Smaller, open-source LLMs show substantial performance gains when augmented with reader-perspective comments, often reaching parity with larger, closed-source models like GPT-4. This makes bias detection more accessible and resource-efficient.
- Comment Selection Matters: While simply appending comments helps, a selector mechanism to filter for useful comments can further enhance performance, though its effectiveness depends on the backbone LLM.
- Comment Length Sensitivity: An increased number of comments can sometimes negatively impact performance, possibly due to increased input length.
Key Results
- The proposed reader-perspective comment augmentation consistently improves F1-scores and accuracy across various LLMs (Phi, Gemma, Llama) on both BASIL and BeyondGender datasets.
- For example, Llama-8B augmented with comments achieved an F1-score of 0.80 for BASIL's "Inf/Lex/non-bias" category, compared to a baseline of 0.61. Similarly, for BeyondGender's "Sexism" category, Llama-8B+AUG reached an F1 of 0.81, up from 0.72.
- Small-sized LLMs like Llama-8B, Phi-3-3.8B, and Gemma-7B showed significant improvements, with some achieving performance comparable to GPT-4.
- The study found that specific comment generation policies (e.g., Policy No. 13 for BASIL, Policy No. 10 for BeyondGender) were more effective than others, highlighting the importance of prompt engineering.
Limitations and Future Work
- Comment Quality Bottleneck: The paper suggests that the quality of generated comments is a key bottleneck. While selectors help, the ultimate performance is tied to how good the initial comments are. Standardized methods for evaluating comment quality are lacking.
- LLM Bias Reflection: Bias inherent in the reader LLMs themselves can be reflected in their generated comments, and bias in the detector LLM can affect interpretation.
- Overcorrection Tendency: LLMs might exhibit overcorrection, leading to decreased performance in some cases.
- Future Directions: The authors suggest developing self-improvement strategies for comment quality and exploring how to mitigate bias in LLMs for fairness and reliability. Further investigation into the precise nature of the relationship between emotion-related comments and bias detection is also warranted.
Summary updated