TL;DR
This paper tackles the challenge of annotator disagreement in hate speech detection by introducing "agreement-aware hate classification." They fine-tune large language models (LLMs) using Direct Preference Optimization (DPO) on this new task, finding that while LLMs are good at majority-based hate classification, they struggle with nuanced agreement prediction, especially without instruction tuning.
What Problem Does This Paper Address?
Subjective tasks, like identifying hate speech, often suffer from annotator disagreement. This means different people can interpret the same text differently, leading to inconsistent labels. Ignoring this disagreement can result in biased models that fail to capture minority viewpoints. Traditional hate speech detection models often rely on majority votes, which can mask these subtle but important disagreements. The paper focuses on the HateWiC (Hateful Word in Context) task, which specifically aims to determine if a word's meaning in a given sentence is hateful. The core problem is how to build models that not only classify hate speech but also account for the varying degrees of agreement among human annotators.
Methods
The authors leverage Large Language Models (LLMs), specifically Llama 3, and fine-tune them using Direct Preference Optimization (DPO). DPO is a more computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF) that aligns LLMs with human preferences by learning from pairwise comparisons of model outputs.
They explore two main approaches:
- Standard Hate Classification: This task aims to predict the majority annotator's judgment on whether a term is hateful.
- Agreement-Aware Hate Classification (Novel Task): This is the paper's key contribution. It aims to predict the proportion of annotators who would classify a term as hateful, using categories like "all," "most," "half," "few," or "none."
They compare two types of Llama 3 models:
- Sft-tuned: Models fine-tuned using supervised learning.
- Instruct-tuned: Models that have undergone further fine-tuning to align with human instructions and preferences.
DPO is applied to both Sft-tuned and Instruct-tuned models for both tasks.
Datasets
The primary dataset used is the HateWiC dataset (Hoeken et al., 2024). This dataset contains approximately 4,000 instances of (non-)hateful terms within example sentences, scraped from Wiktionary. Each instance is annotated by three individuals, providing both majority labels and individual annotator labels. The paper uses the majority labels for Task 1 and individual labels for Task 2. No new benchmark dataset was introduced.
Key Insights and Ideas
- LLMs struggle with nuanced subjectivity: While LLMs are proficient at capturing majority opinions in hate speech classification, they falter when tasked with predicting the distribution of annotator agreement. This highlights the inherent difficulty in modeling subjective human judgment.
- Instruction tuning is crucial for agreement-aware tasks: DPO fine-tuning significantly benefits instruction-tuned models for the agreement-aware task, whereas it offers minimal improvement for standard SFT models. This suggests that instruction tuning primes LLMs to better handle complex, nuanced instructions and subjective signals.
- Prompt formulation matters, but less than subjectivity: Experiments with different prompt variations showed that while prompt structure can influence performance, the core difficulty lies in the inherent subjectivity of the task itself, rather than just how the prompt is phrased.
- "None" class is easier: Models consistently perform better on the "none" (not hateful) class, indicating that identifying clear-cut non-hateful instances is easier than discerning varying degrees of hatefulness or ambiguity.
Key Results
- Task 1 (Majority-based Hate Classification): SFT-tuned models performed best, achieving a macro F1 score of 0.77. DPO fine-tuning showed negligible improvement. Instruct models underperformed, especially on the "hate" class.
- Task 2 (Agreement-Aware Hate Classification): This task proved significantly more challenging. The Instruct-tuned model, after DPO fine-tuning, achieved the best overall performance, with higher accuracy on "all," "most," and "none" classes. However, performance across all models was generally lower than Task 1.
- Model Comparison: Instruct-tuned models, especially when fine-tuned with DPO, showed more balanced predictions and better sensitivity to context-dependent meanings (e.g., the "Oreo" and "bugger" examples in Table 5), suggesting they grasp pragmatic nuance better.
- Error Analysis: Models tend to overpredict the "none" class. The Instruct-tuned model showed a more balanced distribution of predictions compared to SFT variants.
Limitations and Future Work
- Hardware Constraints: The study used smaller, 4-bit quantized models due to hardware limitations. Larger models might yield different results.
- Computational Demands: Optimizing all training components, including hyperparameter tuning, was constrained by computational resources.
- Future Work: The authors suggest further research to systematically explore a broader range of hyperparameter settings and assess performance on larger models. They also emphasize the need for better methods to capture the subjective nature of hate speech detection. The proposed agreement-aware task is intended as a foundation for future advancements.
Summary updated