When a complex machine learning model makes a high-stakes prediction—such as denying a credit application, identifying an operational bottleneck, or flagging a deepfake—developers often deploy post-hoc interpretability tools like SHAP, Layerwise Relevance Propagation (LRP), or causal path interventions. These methods are designed to peek inside the "black box" by calculating mathematical feature weights or generating pixel-level heatmaps.
However, a fundamental paradox has emerged in modern artificial intelligence: the outputs of these interpretability tools are often just as inscrutable to an everyday user as the models themselves. As observed by Sónia Teixeira et al., a model that is technically transparent or mathematically open does not inherently produce explanations that are meaningful, intuitive, or actionable for human decision-makers. This friction reveals a profound disconnect between technical interpretability (how faithfully an explanation mirrors a model's internal mechanics) and human-centric explainability (how effectively a human can understand and act upon that explanation).
The Stakeholder Mismatch
Why does this gap exist? At its core, it is a problem of misaligned audiences. In a large-scale meta-analysis of natural language processing models, Nitay Calderon et al. identified a stark "stakeholder gap." Core AI researchers naturally focus on mechanistic interpretability—isolating individual neurons, circuits, and internal computational pathways to debug, optimize, or audit a system.
Conversely, external stakeholders—such as clinicians, operational managers, lawyers, or end-users—care about functional interpretability. They do not need to know the internal computational architecture of a model; they need to understand high-level input-output relationships and the qualitative reasoning behind a specific decision.
Because existing visualization and interpretability tools primarily target model developers, they focus heavily on low-level technical metrics like gradients and weights (Michelle Peters et al.). As Ramesh Manuvinakurike et al. point out, this leaves domain experts—who possess deep task-specific knowledge but lack advanced coding or statistical skills—unable to interpret why a model failed, preventing them from providing the high-quality feedback necessary to fine-tune and improve the system.
Why Raw Math Fails Human Intuition
To understand why technical explanations fail to build intuition, we must look at what a post-hoc explainer actually produces. Traditional methods yield quantitative importance scores, abstract vector distances, or complex dependency graphs (Ola Ahmad et al.). To a data scientist, a feature weight of $+0.45$ is a clear signal of positive attribution. To a non-expert, however, it functions merely as an additional layer of raw data that requires specialized statistical expertise to interpret (Alina Buss).
Azzurra Ragone et al. emphasize that these raw numerical scores and complex visualizations are fundamentally unintelligible to consumers. For instance, in speech and audio classification, displaying a spectrogram heatmap or phoneme-level data does not translate to an intuitive understanding of how acoustic qualities like tone, pitch, or background noise drove a model's decision (Eliana Pastor et al.).
Crucially, optimizing purely for mathematical rigor can actively degrade human utility. Mark Hamilton et al. demonstrate that ensuring an explanation satisfies formal game-theoretic axioms does not guarantee it will align with human intuition. A mathematically perfect, high-fidelity explanation can easily be too complex or nuanced for a user to apply effectively in a real-world context (Gabriëlle Ras et al.).
Translating Scores into Stories
To bridge this barrier, a growing direction in AI research focuses on transforming raw computational metrics into structured, human-centric narratives. Instead of forcing users to adapt to the language of the machine, these approaches translate the machine's internal state into the language of human reasoning.
1. Mapping Narrative Roles and Social Theories
One powerful strategy involves grounding explanations in established cognitive and social science frameworks. Rather than presenting a list of disconnected feature weights, Or Biran et al. propose mapping feature contributions into qualitative "narrative roles," such as exceptional evidence or missing evidence. This structures the model's internal logic into a justification narrative that mirrors how humans naturally defend their choices. Similarly, Davide Romano et al. leverage social science explanation theories to convert raw feature weights into sparse, actionable feedback tailored to a student's or practitioner's specific background.
2. Conversational and LLM-Driven Translation
Another rapidly expanding paradigm uses Large Language Models (LLMs) as interactive translation layers. Rather than forcing a user to decipher a static heatmap or a raw JSON output, frameworks designed by Sule Tekkesinoglu et al. and Eva Paraschou use LLMs to ingest raw eXplainable AI (XAI) data and synthesize plain-English justifications.
By embedding these systems into conversational interfaces, tools like ECHO (Sebe Vanbrabant et al.) allow users to actively interrogate the model through dialogue. This shifts the user experience from a static, one-way report to an interactive exploration, enabling non-experts to query "Why?" or run counterfactual "What if?" scenarios dynamically.
[ Raw Model Weights ] ──> [ Post-Hoc XAI (SHAP/LRP) ] ──> [ Technical Feature Scores ]
│
▼
[ Human Action ] <── [ Natural Language Narrative ] <── [ LLM Translation Layer ]
Redefining How We Measure Success
If the ultimate goal of explainability is to facilitate human understanding and trust, the metrics we use to evaluate these systems must evolve. Historically, the field has relied on automated, binary metrics of technical faithfulness. However, Wojciech Samek et al. argue that the field critically lacks standardized benchmarks for evaluating cognitive utility—the actual measurable impact an explanation has on human comprehension and decision-making performance.
As Nicholas Halliwell demonstrates, standard technical metrics often fail to capture the qualitative difference between an explanation that is mathematically correct and one that is intuitive to a human. True progress in interpretability requires a shift in perspective: evaluating an explanation not just by how well it mirrors the underlying mathematics of the model, but by how effectively it empowers a human user to make accurate, confident, and safe decisions in the real world.
Key Foundational Work
Spoof Trace Discovery for Deep Learning Based Explainable Face Anti-Spoofing (2015) — contributed 1 concept elements to this direction.
VIKAMINE - Open-Source Subgroup Discovery, Pattern Mining, and Analytics (2012) — contributed 1 concept elements to this direction.