Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models

Esma Balkir, Svetlana Kiritchenko, Isar Nejadgholi, Kathleen FraserNAACL 2022

TL;DR

This paper investigates the current landscape of applying explainable AI (XAI) methods to improve fairness in Natural Language Processing (NLP) models. It highlights a surprising scarcity of research at this intersection and identifies key barriers, such as the local nature of many XAI methods and the difficulty in generalizing insights, hindering their wider adoption for bias mitigation.

What Problem Does This Paper Address?

The increasing deployment of NLP models in critical applications has brought to light their susceptibility to biases, leading to unfair outcomes for certain demographic groups. While explainable AI (XAI) methods are often touted as a solution for detecting and mitigating these biases by providing insights into model behavior, the practical application of XAI for fairness in NLP remains underdeveloped. This paper addresses the gap between the theoretical promise of XAI for fairness and its limited practical implementation in NLP, exploring why this intersection is not as widely researched as one might expect and identifying the challenges that prevent broader adoption.

Methods

The paper provides a review of existing XAI methods relevant to NLP, categorizing them based on whether they offer local or global explanations and whether they are self-explaining or post-hoc. Key methods discussed include:

  • Feature Attribution Methods: Techniques like Gradients, Integrated Gradients, LIME, and SHAP that aim to determine the importance of individual tokens in an input for a model's prediction.
  • Extractive Rationales: Methods that identify specific snippets of input text that trigger a prediction.
  • Counterfactual Explanations: Approaches that generate new instances by minimally altering an input to change the model's output, helping to understand what would need to change for a different outcome.
  • Influence Functions: Methods that pinpoint training data examples most influential for a particular prediction.
  • Concept-based Explanations: Techniques that use higher-level semantic concepts to explain model behavior.
  • Rule-based Explanations: Methods like Anchors that generate explanations in the form of rules.

The paper also discusses fairness metrics, distinguishing between group fairness (performance disparities across groups) and individual fairness (consistent behavior for similar examples).

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Scarcity of Research: Despite the strong motivation for XAI to address fairness, the authors found a surprisingly small number of studies that actually apply XAI methods to uncover or investigate biases in NLP models.
  • Limited Scope: The existing applications of XAI for fairness in NLP are concentrated on a limited number of tasks (primarily hate speech detection) and often use only a few examples for illustration.
  • Local vs. Global Explanations: Many XAI methods provide local explanations (for individual instances), which makes it difficult to generalize insights to understand systematic model bias. Identifying fairness problems often requires understanding global behavior, which local explanations struggle to provide.
  • Generalization Challenges: Even when a local XAI method identifies a potential bias, it's hard to determine the extent to which this insight can be generalized to other inputs or the model as a whole.
  • Surface-Level Bias Focus: Current XAI applications for fairness often focus on identifying token-level correlations that are surface-level or non-causal. However, some biases are systemic and deeply embedded, requiring more than just token-level analysis.
  • Fairwashing Risk: There's a concern that XAI methods could be used to "fairwash" biased models by adversarially manipulating explanations to obscure the model's reliance on protected attributes.
  • Procedural vs. Outcome Fairness: The paper highlights a conceptual gap: while Fair ML literature primarily focuses on outcome fairness (fair results), XAI is often motivated by procedural fairness (understanding the model's internal reasoning).

Key Results

The paper's primary "result" is a comprehensive review and analysis of the current state of XAI application in NLP fairness. It reveals that:

  • Only a handful of NLP papers apply XAI methods to uncover or investigate biases.
  • These applications predominantly use feature attribution methods.
  • Most studies employ these methods for demonstration purposes on a limited number of examples.
  • The focus is heavily on hate speech detection, with a narrow range of bias types being investigated.

Limitations and Future Work

Limitations:

  • Local Nature of XAI: The local nature of most XAI methods makes it difficult to identify and generalize fairness issues without prior knowledge of what biases to search for.
  • Generalizability: Insights from local explanations are hard to generalize, hindering the application of mitigation strategies.
  • Surface-Level Bias: XAI methods are primarily effective for surface-level, token-level spurious correlations, but may struggle with deeper, systemic biases.
  • Human Recognition: XAI relies on humans to recognize biases, but some biases can be subtle and difficult for humans to identify, even with explanations.
  • Fairwashing: The potential for explanations to be manipulated to obscure bias is a significant concern.

Future Work:

  • Principled Aggregation: Develop methods to aggregate local explanations into more global and actionable insights.
  • Global Explainability: Explore global explainability methods that can uncover unknown biases.
  • Beyond Token-Level: Investigate methods to uncover biases that are not surface-level or token-level correlations, potentially requiring analysis of model structure and training.
  • Causal Reasoning: Further explore the link between causality, robustness, and fairness, as causal correlations are more invariant across distributions.
  • Human-Subject Studies: Conduct more studies with human subjects to understand how well they can recognize unintended biases through XAI methods.
  • Procedural Fairness Metrics: Develop more precise quantitative metrics for procedural fairness to guide XAI development for fairer ML models.
  • Certifiably Faithful Explanations: Extend certifiable robustness to explainability methods to ensure explanations accurately reflect model behavior.

Summary updated

BibTeX
@inproceedings{balkir2022challenges,
  author = {Esma Balkir and Svetlana Kiritchenko and Isar Nejadgholi and Kathleen Fraser},
  title = {{Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure