Introduction: The Challenge of Unintended Offense
Traditional offensive language detection models primarily rely on "intended affective markers"—explicit slurs, hashtags like #hate, or known toxic keywords. While effective for blatant toxicity, this approach fails to capture a significant portion of human social friction: unintended offense. These are messages where the sender may believe they are being helpful or neutral, but the receiver perceives the statement as offensive or a microaggression.
Because these offenses are context-dependent and lack explicit "bad words," standard models struggle to identify them. This paper addresses the gap by introducing a novel dataset built around "conflict cues" in conversation threads, capturing the mismatch between a sender's intent and a receiver's perception.
Methodology: Conflict-Based Data Collection
The researchers developed a framework that works backward from conflict resolution. Instead of filtering for keywords, they identified "response cues"—specifically the phrase "Didn't mean to offend you"—which signals a perceived offense and a subsequent attempt at clarification.
The data collection pipeline involved:
- Response Cue Identification: Searching for specific apology/clarification phrases.
- Conversation Reconstruction: Pulling full Twitter threads to provide context.
- Multi-Stage Filtering: Applying quotation, ambiguous pronoun, and conversation dynamic filters to ensure high-quality, direct two-person interactions.
- Human Annotation: Using Amazon Mechanical Turk to rate "target posts" on a 0–100 offensiveness scale, evaluating them both in isolation and with surrounding context.
Dataset Characteristics and Implicitness
The resulting Unintended Offense Dataset consists of 4,027 fine-grained conversation threads. A key finding is the dataset's high degree of "implicitness." Compared to standard benchmarks like OLID or Waseem, this dataset captures offenses that are significantly more subtle and context-reliant.
The study also observed a "Context Escalation Effect": providing context often increased the perceived offensiveness of a post, as neutral-sounding sentences became biting when viewed as responses to specific prompts. This highlights the necessity of conversational context in modern moderation tools.
Key Results and Model Performance
Experiments revealed that standard BERT models trained on traditional datasets (like OLID or Founta) performed poorly on this new data, often failing to achieve a Macro F1 score above 0.5. However, when BERT was trained specifically on the Unintended Offense data, performance improved significantly to a Macro F1 of 0.843.
Large Language Models (LLMs) like GPT-4 also struggled in zero-shot scenarios, frequently misclassifying offensive posts as neutral. Performance only reached acceptable levels (Macro F1 of 0.728) when provided with "few-shot" examples from the new dataset, highlighting the difficulty of detecting subtle social friction without specialized training data.
Limitations and Future Directions
While the dataset provides a new lens for toxicity detection, it is limited by the inherent subjectivity of offense and its focus on English-language Twitter. Future work may explore these dynamics across different platforms (e.g., Reddit, LinkedIn) and languages, or move beyond binary classification toward more nuanced regression-based models using the 0–100 scale provided in the dataset.
Summary updated