Abstract
In Explainable AI (XAI), counterfactual explanations (CEs) are a well-studied method to communicate feature relevance through contrastive reasoning of "what if" to explain AI models' predictions. However, they only focus on important (i.e., relevant) features and largely disregard less important (i.e., irrelevant) ones. Such irrelevant features can be crucial in many applications, especially when users need to ensure that an AI model's decisions are not affected or biased against specific attributes such as gender, race, religion, or political affiliation. To address this gap, the concept of alterfactual explanations (AEs) has been proposed. AEs explore an alternative reality of "no matter what", where irrelevant features are substituted with alternative features (e.g., "republicans" -> "democrats") within the same attribute (e.g., "politics") while maintaining a similar prediction output. This serves to validate whether AI model predictions are influenced by the specified attributes. Despite the promise of AEs, there is a lack of computational approaches to systematically generate them, particularly in the text domain, where creating AEs for AI text classifiers presents unique challenges. This paper addresses this challenge by formulating AE generation as an optimization problem and introducing NoMatterXAI, a novel algorithm that generates AEs for text classification tasks. Our approach achieves high fidelity of up to 95% while preserving context similarity of over 90% across multiple models and datasets. A human study further validates the effectiveness of AEs in explaining AI text classifiers to end users.
Introduction
The prevailing paradigm in Explainable AI (XAI) has long been the Counterfactual Explanation (CE). A CE identifies the minimal change required to flip a model's prediction, answering the question: "What if this feature were different?" While CEs are effective at highlighting influential features, they offer no insight into irrelevant ones. In high-stakes applications like content moderation or hiring, understanding what does not affect a model is just as critical as understanding what does. For instance, we must ensure that a toxicity classifier's decision is independent of protected attributes like race or gender.
To address this, we formalize Alterfactual Explanations (AEs) for text. An AE follows the logic of "No matter what": "No matter what the gender of the subject is, the classifier still predicts the same outcome." By substituting irrelevant features with alternatives from the same semantic field, AEs provide a tool for auditing model fairness and robustness.
Methodology
We introduce NoMatterXAI, a greedy optimization algorithm designed to generate AEs that are semantically distinct from the original input while maintaining the model's prediction and the overall context. The generation process consists of two primary stages:
- Irrelevant Feature Selection: The algorithm ranks words by their predictive importance and targets those with the lowest impact for modification.
- Feature Perturbation: Target words are replaced with their linguistic "opposites" to maximize the distance from the original input while staying within the same semantic category. We utilize two strategies: ConceptNet for structured knowledge retrieval (e.g., "red" to "blue") and LLMs (GPT-3.5 Turbo) for context-aware antonym generation.
To ensure grammatical and semantic integrity, the algorithm includes a Double Negative Detection constraint, preventing changes that would inadvertently invert the sentence's meaning.
Experimental Results
NoMatterXAI was evaluated across four text classification tasks: Gender Bias (GB), Hate Speech (HS), Emotion (EMO), and Jigsaw Toxicity (JIG), using DistilBERT, BERT, and RoBERTa models. The algorithm achieved a Fidelity of up to 95% and maintained over 90% semantic similarity to the original text.
A key insight from our experiments is that Fidelity serves as a proxy for model bias. As shown in the figure below, the fidelity of generated AEs correlates strongly with empirical bias. If a model is biased against a specific attribute, the algorithm will struggle to find valid AEs for that attribute because changing it would likely flip the model's prediction.
Furthermore, a human study with 149 participants demonstrated that users are significantly better at identifying biased models when presented with NoMatterXAI's alterfactual explanations compared to baseline methods.
Conclusion and Future Work
NoMatterXAI introduces a systematic approach to generating "no matter what" explanations, filling a critical gap in the XAI landscape. By focusing on irrelevant features, it provides a powerful mechanism for auditing black-box models for bias. Future research will focus on reducing the computational cost of the greedy search and expanding the knowledge bases to handle more complex linguistic structures and multi-sentence documents.
Summary updated