If there's a Trigger Warning, then where's the Trigger? Investigating Trigger Warnings at the Passage Level

Matti Wiegmann, Jennifer Rakete, Magdalena Wolska, Benno Stein, Martin PotthastarXiv 2024 · arXiv

This paper, recently published by researchers from Bauhaus-Universität Weimar and other institutions, tackles a practical but often overlooked question in content moderation: If a document has a trigger warning, where exactly is the "trigger"?

TL;DR

The authors move beyond document-level trigger warnings to investigate the feasibility of identifying specific "triggering passages" within a text. They introduce a new dataset of 4,135 annotated passages and demonstrate that while automatic detection is feasible, the task is inherently subjective, with performance varying significantly between fine-tuned models and few-shot LLMs.

What Problem Does This Paper Address?

Trigger warnings (TWs) are typically applied at the document level. While helpful, this "coarse-grained" approach has two major drawbacks. First, it doesn't tell the reader where the sensitive content is, potentially preventing them from reading large portions of perfectly safe text. Second, for developers of safety tools, document-level labels provide a weak signal for training models to understand why a text is harmful.

The authors investigate "fine-grained" trigger detection at the passage level. They explore whether humans can reliably agree on what constitutes a trigger and whether modern NLP models can automate this identification.

Methods

The researchers employed a multi-stage approach:

  1. Candidate Retrieval: They used keyword-based retrieval (BM25) to pull five-sentence passages from a large corpus of fan fiction (WTWC-22) that were likely to contain one of eight specific triggers.
  2. Human Annotation: Each passage was labeled by three annotators. To handle the subjectivity of the task, they provided annotators with a specific "Persona" and "Definition" for each trigger to ground their decisions.
  3. Modeling: They compared two main paradigms:
  4. Fine-tuning: They took a RoBERTa-base model, further pre-trained it on fan fiction, and fine-tuned it on their new dataset using various strategies (Binary, Multiclass, Multilabel).
  5. Few-shot Prompting: They tested several Large Language Models (LLMs) including GPT-4, Mixtral 8x7B, and Llama 2, providing them with a few examples (demonstrations) in the prompt.
  6. Evaluation: They tested models on both "In-Distribution" (ID) data and "Out-of-Distribution" (OOD) data, where the OOD set used keywords unseen during training to test the models' conceptual understanding.

Datasets

The study introduces a new benchmark derived from the Webis Trigger Warning Corpus 2022 (WTWC-22).

  • Size: 4,135 English passages (5 sentences each).
  • Labels: Eight common trigger categories: Violence, Death, War, Abduction (Aggression category); and Racism, Homophobia, Misogyny, Ableism (Discrimination category).
  • Structure: Each passage has three binary human labels.

Detailed statistics on the dataset composition can be found in the paper's summary table:

Descriptive dataset statistics including passage counts and inter-annotator agreement.

Key Insights and Ideas

  • Trigger Warning Detection is Subjective: The inter-annotator agreement (Krippendorff’s $\alpha$) was 0.35, which is considered "fair." This confirms that what one person finds triggering, another might find merely descriptive or even "enticing" (in the context of fiction).
  • The "Lexical Gap" in Fine-tuning: Fine-tuned models performed well on ID data but struggled on OOD data. They tended to overfit to specific "slurs" or keywords rather than learning the broader concept of the trigger.
  • LLMs are More Sensitive: Few-shot LLMs generally had a much higher "positive rate" than fine-tuned models, suggesting they are more cautious (or sensitive) when identifying potential triggers.
  • Context Matters: The authors found that many misclassifications occurred in "edge cases" where the triggering concept was mentioned in the background or in a fantastical setting (e.g., "undead" characters in a fantasy novel).

Examples of these nuances and how they lead to model disagreement are illustrated here:

Figure 3: Trigger Warning. Selected example passages with assigned warning and number of positive votes. The center sentence that was retrieved via keywords is is italics.

Key Results

  • Top Performance: The best-performing model was a Multiclass fine-tuned model on in-distribution data, achieving an accuracy of 0.82.
  • Generalization: On out-of-distribution data, Mixtral 8x7B was the most robust, showing that LLMs are better at capturing the concept of a trigger when specific training keywords are missing.
  • Category Variance: "Aggression" triggers (like Violence) were generally easier for models to identify than "Discrimination" triggers (like Misogyny), which often rely on more subtle social cues.

Limitations and Future Work

The authors acknowledge several constraints:

  • Demographic Bias: Annotators were primarily computer science students, which may not represent the sensitivities of the general population or those clinically affected by trauma.
  • Scope: The study only covered 8 out of 36+ known trigger categories.
  • The Personalization Frontier: Because trigger sensitivity is so individual, the authors suggest that personalized trigger warning assignment—where a model learns a specific user's thresholds—is the most promising future direction for this research.

Summary updated

Abstract

Trigger warnings are labels that preface documents with sensitive content if this content could be perceived as harmful by certain groups of readers. Since warnings about a document intuitively need to be shown before reading it, authors usually assign trigger warnings at the document level. What parts of their writing prompted them to assign a warning, however, remains unclear. We investigate for the first time the feasibility of identifying the triggering passages of a document, both manually and computationally. We create a dataset of 4,135 English passages, each annotated with one of eight common trigger warnings. In a large-scale evaluation, we then systematically evaluate the effectiveness of fine-tuned and few-shot classifiers, and their generalizability. We find that trigger annotation belongs to the group of subjective annotation tasks in NLP, and that automatic trigger classification remains challenging but feasible.
BibTeX
@misc{wiegmann2024if,
  author = {Matti Wiegmann and Jennifer Rakete and Magdalena Wolska and Benno Stein and Martin Potthast},
  title = {{If there's a Trigger Warning, then where's the Trigger? Investigating Trigger Warnings at the Passage Level}},
  year = {2024},
  eprint = {2404.09615},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.09615},
}

Loading related content...

Figure