Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection

Esma Balkir, Isar Nejadgholi, Kathleen C. Fraser, Svetlana KiritchenkoNAACL 2022 · DOI

Abstract

Explainability in AI often relies on feature attribution, but standard methods frequently fail to distinguish between whether a word is necessary for a prediction or sufficient on its own. This paper introduces a novel model-agnostic method that splits importance into these two theoretically grounded scores. By leveraging a generative infilling model to create realistic perturbations, the authors diagnose specific biases in hate speech classifiers that traditional methods like SHAP or LIME overlook.

Introduction

Hate speech detection is plagued by "spurious correlations," where models flag neutral sentences simply because they contain specific identity terms like "Muslim" or "women." Traditional feature attribution methods assign a single importance score to words, which is often ambiguous. Does high importance mean the model requires that word to justify a "hateful" label, or does it mean the word itself is enough to trigger the label regardless of context?

Furthermore, common perturbation techniques like masking or deletion create "out-of-distribution" examples. These nonsensical inputs can cause classifiers to behave erratically, leading to unfaithful explanations. This study addresses these issues by defining necessity and sufficiency and using generative infilling to maintain data distribution.

Methodology

The proposed method calculates two complementary scores for any given word in a sentence:

  1. Necessity: Measures if changing a specific word flips the prediction from "hateful" to "non-hateful." High necessity indicates the model relies on this word to justify its classification.
  2. Sufficiency: Measures if keeping the word while changing the rest of the sentence maintains the "hateful" prediction. High sufficiency for an identity term is a strong indicator of model bias.

To ensure realistic perturbations, the authors use an Infilling Language Model (ILM)—a fine-tuned GPT-2—to replace masked words with fluent alternatives. Crucially, the ILM is trained only on neutral data. This allows the researchers to test if a classifier incorrectly triggers a "hate" label when an identity term is placed in a benign, realistic context.

The pipeline for calculating necessity and sufficiency using masking and generative infilling.

Experimental Setup

The researchers evaluated six BERT-based classifiers trained on three major datasets: Founta2018, Davidson2017, and Vidgen2021. The ILM was trained on a combined dataset of ~130,000 neutral instances from various sources including Wikipedia and Reddit. Evaluation was conducted using HateCheck, a functional test suite of hand-crafted templates (e.g., "I hate [IDENTITY]") designed to probe specific model weaknesses.

Analysis and Results

The dual-score approach provides significant diagnostic power. For instance, models trained on broader "abuse" labels often showed lower necessity for identity terms compared to those trained strictly on "hate" labels, suggesting that "abuse" detection relies more on toxic keywords than on identifying a target.

A key finding was the "over-sensitivity" of models to specific identity terms. In qualitative visualizations, the word "Muslims" often exhibited near-total sufficiency for a "hateful" prediction, whereas "women" did not, exposing a clear group-based bias.

Comparison of necessity (orange) and sufficiency (blue) scores for "women" vs "Muslims" in a biased classifier.

Compared to traditional methods like LIME, which often gave inconsistent or low importance to terms causing false positives, the necessity/sufficiency framework consistently identified these biases. By staying on the "data manifold" via generative infilling, the explanations remain faithful to the model's real-world behavior.

Discussion and Conclusion

While powerful, the method is computationally intensive due to the need for numerous generative perturbations per word. Future work could extend this framework to multi-class problems and integrate these metrics into the training loop to help developers proactively de-bias models. By distinguishing between necessity and sufficiency, researchers can move beyond simple "importance" and gain a deeper understanding of why AI models make the decisions they do.

Summary updated

BibTeX
@inproceedings{balkir2022necessity,
  author = {Esma Balkir and Isar Nejadgholi and Kathleen C. Fraser and Svetlana Kiritchenko},
  title = {{Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.192},
}
Figure