TL;DR
When machine learning models are biased, making their explanations easier to understand actually decreases user trust. This paper introduces a comprehensive taxonomy of explainability visualization design choices and proves that clearer explanations expose underlying model biases, which naturally and appropriately down-regulates user trust.
What Problem Does This Paper Address?
Explainable AI (XAI) tools like SHAP, LIME, and ELI5 are widely used to demystify "black-box" machine learning models. However, we have a poor understanding of how the visual design of these explanations affects a non-expert user's comprehension, their ability to spot algorithmic bias, and their ultimate trust in the system.
If a model is fundamentally unfair—such as discriminating against loan applicants based on gender—an effective visualization should help a user detect that flaw rather than foster blind trust. The authors bridge this gap by systematically analyzing how specific visualization design characteristics influence the delicate interplay between comprehension, bias perception, and trust.
Methods
The authors approached this problem in three distinct phases:
- Taxonomy Development: Following established iterative methodologies, they analyzed 26 local explainability visualizations to build a taxonomy consisting of 54 design characteristics across 27 dimensions (e.g., whether feature impact magnitude is explicit or implicit, how color is utilized, and whether alternative inputs are displayed).
- Correlational User Study: They conducted an online study with 438 non-expert participants on Prolific. Participants evaluated a loan-approval model using visualizations from six state-of-the-art tools: SHAP Waterfall, SHAP Force, LIME, ELI5, Ceteris-Paribus (CP) Profiles, and Anchors.
- Causal Experiments: To prove causal relationships, they ran follow-up experiments with an additional 380 participants, explicitly manipulating:
- Explicitness: Adding explicit mouse-over values to an implicit visualization (CP Profiles).
- Fairness: Evaluating a truly fair model versus a biased model using identical visual layouts.
- Bias Perception: Modifying visual complexity (e.g., removing axes or adding alternative outputs) to artificially shift bias perception while holding objective comprehension steady.
Datasets
The authors utilized the classic Census Income dataset (also known as the Adult dataset), which contains demographic and income features for 48,842 individuals. Because this historical dataset features a severe gender imbalance (loans are granted to 31% of men but only 11% of women), an unconstrained LightGBM classifier trained on it inherently picked up these sexist biases.
For their fairness experiments, the authors used the Seldonian Toolkit to train a fair random forest model constrained to enforce demographic parity across sex and age.
Key Insights and Ideas
- The Paradox of Comprehension: There is a striking inverse relationship between how well a user understands a biased model and how much they trust it.
- The Mediation Effect: This negative correlation is entirely driven by bias perception. Clearer, more explicit visualizations make the model's underlying unfairness obvious. Once users see the bias, their trust drops.
- Explicitness Drives Understanding: Visualizations that explicitly print out feature impact directions and numerical magnitudes (like LIME) yield significantly higher objective comprehension than those requiring users to infer impacts from curves or rules (like CP Profiles or Anchors).
- Behavioral Alignment Blinds Users: If a participant believes the model's decision would personally benefit them, their perception of bias plummets, and their trust increases—regardless of whether the model is actually fair.
Key Results
- Tool Performance: LIME achieved the highest objective comprehension scores among participants, while Anchors and CP Profiles scored the lowest.
- Mediation Metrics: Statistical modeling confirmed that a 1-unit increase in objective comprehension increased perceived bias by 0.27 units ($p < 0.001$). In turn, a 1-unit increase in perceived bias decreased trust by 2.35 units ($p < 0.001$).
- Causal Impact of Explicitness: Transforming an implicit visualization into an explicit one significantly boosted comprehension (Cohen's $d = 0.68$) and bias perception ($d = 0.22$), while causing a direct drop in trust ($d = -0.26$).
- Manipulating Trust via Design: The authors proved that they could successfully manipulate user trust without changing the underlying model. By visually streamlining explanations to obscure bias, they artificially engineered a small but significant increase in user trust ($d = 0.20$).
Limitations and Future Work
- Simplified Feature Space: To prevent participant fatigue, the study restricted the model to just 5–6 features. Real-world models handle hundreds of variables where biases are much harder to isolate.
- Binary Gender Constraints: The dataset forced a binary classification of sex. Interestingly, all 19 non-binary participants in the study flagged both the biased and the "fair" models as discriminatory, highlighting a critical need for XAI research that explicitly accounts for non-binary and intersectional demographic realities.
- Isolated Explanation Types: The study looked at tools in isolation. Future work should investigate whether combining different explanation styles (e.g., pairing additive feature contributions with counterfactual examples) can further improve bias detection without overloading the user.
Summary updated