STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions

Robert Morabito, Sangmitra Madhusudan, Tyler McDonald, Ali EmamiEMNLP 2024 · DOI

Executive Summary

The STOP (Sensitivity Testing on Offensive Progressions) dataset introduces a novel benchmark for evaluating bias in Large Language Models (LLMs). Unlike traditional benchmarks that focus on isolated incidents, STOP assesses how models handle escalating microaggressions. Research indicates that while current models struggle to identify the "tipping point" of inappropriate behavior, fine-tuning on human-aligned sensitivity scores significantly enhances performance on standard bias benchmarks like BBQ and StereoSet.

The Challenge of Contextual Bias

Most existing bias benchmarks, such as BBQ or CrowS-Pairs, evaluate language in a vacuum by presenting single scenarios to identify stereotypes. However, real-world bias is dynamic, often manifesting as a series of subtle microaggressions that escalate from implicit to explicit harm.

Current evaluation methods often fail to capture this contextual evolution. A model might flag a blatant slur but miss a pattern of discriminatory behavior—for instance, a store clerk following a specific customer across multiple aisles. Identifying the "tipping point" where a situation transitions from appropriate to inappropriate is essential for developing safer, human-aligned AI.

Methodology: The STOP Framework

The researchers developed the Sensitivity Testing on Offensive Progressions (STOP) framework to capture these nuances.

  1. Formalization: Each entry consists of a "progression" of five consecutive sentences. As the narrative unfolds, the level of bias or offensive content increases.
  2. The Task: Models are presented with sentences sequentially and asked to determine if the situation remains "appropriate."
  3. Counterfactuals: To test robustness, "counterfactual" sentences provide context that attempts to justify problematic behavior. This evaluates whether a model can maintain an ethical stance despite seemingly logical excuses for bias.
  4. Construction: A "human-in-the-loop" approach was employed. After writing 90 "seed" progressions, GPT-4 scaled these across 46 sub-demographics. Finally, humans validated every entry and created the counterfactuals.
The STOP dataset creation workflow, illustrating the transition from concept to testing.

Dataset Comparison

The STOP dataset comprises 450 offensive progressions and 2,700 unique sentences covering 9 major demographics (e.g., Race, Religion, Gender) across three severity levels. Compared to existing benchmarks, STOP offers broader coverage in areas like Political Ideology and Class.

Comparison of STOP against BBQ, CrowS-Pairs, and StereoSet across various demographic categories.

Key Insights

  • The Tipping Point Inconsistency: Models exhibit varying sensitivity. Some are "over-sensitive," flagging harmless interactions, while others are "under-sensitive," missing clear microaggressions.
  • Human vs. Machine Perception: Humans achieved 100% accuracy on explicit bias but were often more permissive than models in low-severity scenarios. This suggests that current AI safety filters may be more stringent than typical human judgment for subtle biases.
  • Alignment via Progression: Fine-tuning Llama 3-70b to match human sensitivity scores on these progressions improved its performance on the STOP task and made it more accurate and "brave" (less prone to unnecessary refusals) on downstream benchmarks.

Results and Performance

  • Top Performers: Llama 2-70b achieved a success rate of 69.8%, outperforming GPT-4 (57.8%) in identifying offense thresholds.
  • Fine-tuning Impact: Fine-tuning Llama 3-70b on STOP human responses led to a 191% increase in its "answer rate" on the StereoSet benchmark, reducing over-refusal.
  • Consistency: Llama 3-8b showed the most consistent judgment across severity levels, whereas Gemma-7b struggled, particularly in moderate scenarios.

Limitations and Future Directions

While the 2,700-sentence dataset is a significant step, manual crafting limits its scope. Future iterations could utilize automated, adversarial generation. Additionally, since "appropriateness" is culturally subjective, expanding the human annotator pool globally would enhance the baseline. Currently an evaluation tool, STOP's insights could eventually be integrated into initial pre-training or RLHF phases.

Summary updated

BibTeX
@inproceedings{morabito2024stop,
  author = {Robert Morabito and Sangmitra Madhusudan and Tyler McDonald and Ali Emami},
  title = {{STOP! Benchmarking Large Language Models with Sensitivity Testing on Offensive Progressions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.243},
}
Figure