Explainable AI (XAI) has a foundational problem: we have built an immense library of tools to peek inside black-box models, but we lack a shared language to describe what they actually do. When one researcher claims a model is "interpretable," another might call it "comprehensible," while a third evaluates its "faithfulness." As David Bau et al. observe in Explaining Explanations: An Overview of Interpretability of Machine Learning, terms like interpretability and explainability are routinely used interchangeably, creating a conceptual overlap that prevents us from establishing rigorous auditing frameworks.
This isn't just a pedantic debate over semantics. The lack of standardized definitions—what Sebastian Palacio et al. call a "Tower of Babel" problem in XAI Handbook: Towards a Unified Framework for Explainable AI—means that different sub-fields are effectively building isolated silos. Derek Doran et al. demonstrate in What Does Explainable AI Really Mean? A New Conceptualization of Perspectives that without clear operational boundaries, the community applies these concepts in vague, quasi-mathematical ways. The consequence is a fragmented landscape where practitioners cannot easily match a specific explanation method to their data type, model architecture, or safety requirements (Gabriëlle Ras et al. in Explainable Deep Learning: A Field Guide for the Uninitiated).
Deconstructing Explanations into Mathematical Primitives
To move past this fragmentation, researchers are shifting away from proposing new ad-hoc explanation algorithms and are instead focusing on meta-frameworks that deconstruct existing methods into shared mathematical primitives. The intuition here is simple: if we can show that seemingly disparate algorithms are actually doing the same underlying operations, we can compare them directly.
Consider model-agnostic explanations like LIME or Shapley values. On the surface, they look entirely different. However, Christian A. Scholbeck et al. introduce the SIPA framework in Sampling, Intervention, Prediction, Aggregation: A Generalized Framework for Model-Agnostic Interpretations, proving that almost all model-agnostic interpretations follow a strict four-stage pipeline:
- Sampling data points around the instance of interest.
- Intervention on those samples (such as masking features).
- Prediction using the black-box model on the modified samples.
- Aggregation of the predictions into a human-readable summary.
By standardizing this notation, the authors provide a common language that exposes the exact mathematical relationships between previously isolated techniques.
Similarly, Stephen E. Wormald et al. propose the Qi-Framework (Question-Information) in Abstracting general syntax for XAI after decomposing explanation sub-components. By decomposing 184 distinct XAI methods, the authors abstract a unified syntax that defines exactly what information is being extracted and how it serves human understanding.
Going even deeper into the mathematical foundations, Pietro Barbiero et al. argue in Actionable Interpretability Must Be Defined in Terms of Symmetries that traits like sparsity or modularity should not be treated as isolated goals. Instead, they ground interpretability in category theory and structure-preserving transformations (symmetries), offering a unified, testable recipe for evaluating whether a model's internal mechanics align with human concepts.
Resolving the Benchmarking Crisis
The fragmentation of XAI extends heavily into how we evaluate explanations. Historically, researchers relied on qualitative "sanity checks" (e.g., looking at a single saliency map and deciding it "looks right"). When quantitative metrics are used, the field suffers from severe semantic fragmentation. As Phuong Le reveals in Benchmarking eXplainable AI - A Survey on Available Toolkits and Open Challenges, different software libraries frequently use entirely different names for the exact same underlying mathematical evaluation metric, leading to inconsistent and non-reproducible results.
To establish objective evaluation standards, recent work focuses on creating centralized, open-source benchmarking ecosystems. For example, Chirag Agarwal et al. developed OpenXAI: Towards a Transparent Evaluation of Model Explanations, which introduces standardized datasets, unified quantitative metrics, and public leaderboards to systematically evaluate post-hoc explanations. In parallel, Mohamed Karim Belaid et al. introduce Compare-xAI: Toward Unifying Functional Testing Methods for Post-hoc XAI Algorithms into a Multi-dimensional Benchmark, which applies a suite of functional "unit tests" across dimensions like fidelity and stability to compute a unified comprehensibility score.
This push for standardization is also transforming specialized sub-fields of interpretability. In the domain of counterfactual explanations—which explain a model's decision by showing what minimal changes would alter the outcome—Junqi Jiang et al. introduce RobustX: Robust Counterfactual Explanations Made Easy to provide a unified framework for evaluating explanation robustness against model fine-tuning. Meanwhile, Eleonora Poeta et al. provide a systematic taxonomy for concept-based explanations in Concept-based Explainable Artificial Intelligence: A Survey, helping practitioners navigate the trade-offs between different algorithmic approaches.
By moving from isolated, ad-hoc implementations (a challenge highlighted early on by Maximilian Alber et al. in iNNvestigate neural networks!) toward unified mathematical syntaxes and standardized evaluation ecosystems, the field of XAI is transitioning from an art of intuitive design into a rigorous, comparative science.
Key Foundational Work
What Does Explainable AI Really Mean? A New Conceptualization of Perspectives (2017) — contributed 1 concept elements to this direction.
Explaining Explanations: An Overview of Interpretability of Machine Learning (2018) — contributed 1 concept elements to this direction.
iNNvestigate neural networks! (2018) — contributed 1 concept elements to this direction.