TL;DR
This paper bridges the gap between academic Explainable AI (XAI) research and its industrial application by mapping XAI needs across the entire Machine Learning life cycle. Through 36 stakeholder interviews, the authors reveal that while research is heavily focused on model evaluation for data scientists, industrial reality demands XAI tools for data preparation, monitoring, and non-expert users.
What Problem Does This Paper Address?
Despite the explosion of AI in industry, moving from a "Proof of Concept" to a deployed, reliable system remains a significant hurdle. A primary culprit is the "black-box" nature of modern models, which hampers trust, complicates debugging, and makes regulatory compliance difficult.
The authors argue that current XAI research is too narrow. Most academic papers focus on the "Evaluation" phase—explaining why a model made a specific prediction after it has already been trained. However, industrial practitioners need "explainability" at every step: from understanding data biases during collection to performing root-cause analysis when a model's performance drifts in production. There is a fundamental mismatch between the tools academia is building and the practical needs of the "producing industries" (e.g., manufacturing and automation).
Methods
The researchers conducted a qualitative study consisting of 36 semi-structured interviews with a diverse range of roles, including data scientists, ML researchers, managers, safety engineers, and service technicians from Siemens AG, startups, and technical associations.
To organize their findings, they used the CRISP-ML (Cross-Industry Standard Process for Machine Learning) framework, which divides the ML life cycle into six stages:
- Business and Data Understanding
- Data Preparation
- Modelling
- Evaluation
- Deployment
- Monitoring and Maintenance
By mapping interview insights against current academic literature for each stage, they identified "alignments" (where research meets needs) and "mismatches" (where industry is left wanting).
Datasets
As this is a qualitative research paper focused on industrial practice and literature review, it does not introduce or use standard ML training datasets (like ImageNet or GLUE). The primary "data" consists of the transcribed protocols from the 36 expert interviews and a survey of existing XAI literature.
Key Insights and Ideas
- XAI as a Competitive Advantage: In the industrial market, explainability is no longer just a "nice-to-have." It is becoming a "distinguishing factor" that companies use to build trust with customers who are increasingly AI-mature and skeptical of opaque systems.
- The "Expert" Trap: Most XAI tools (like SHAP or LIME) are designed by data scientists for data scientists. In reality, the people who need explanations are often domain experts (e.g., a mechanical engineer) or technicians who lack deep ML knowledge. The paper highlights a desperate need for XAI to speak the "language of the domain."
- The Neglected "Monitoring" Phase: While academia loves the "Evaluation" phase, industry is deeply concerned with "Monitoring and Maintenance." When a model fails in a factory, technicians need XAI to tell them why it failed and how to fix it (e.g., "the sensor is dusty" vs. "the data distribution has shifted").
- Scalability is a Dealbreaker: Many popular XAI methods are computationally expensive. The authors found that the high cost of calculating explanations (like Shapley values) often prevents their use in real-time industrial deployment.
Key Results
The paper provides a comprehensive "map" (Figure 2) of XAI across the ML life cycle. Key qualitative findings include:
- Data Collection: XAI is being used to detect "wrong labels" and data bias, though academic tools for this (like influence functions) are not yet popular in practice.
- Modelling: There is a strong desire for "XAI-guided" development, such as using explanations to choose hyperparameters or network architectures, rather than just evaluating the final result.
- Regulation: The upcoming EU AI Act is a major driver. High-risk AI applications will likely require transparency and explainability by law, making XAI a mandatory component of the certification process.
- Human-Centricity: The study confirms that "Visual Analytics" and human-computer interaction (HCI) are just as important as the underlying algorithms. An explanation is useless if the end-user can't interpret it.
Limitations and Future Work
The authors acknowledge that their sample size (36 interviews) is relatively small and focused primarily on the "producing industries," meaning the results might differ for sectors like finance or healthcare.
Future Work should focus on:
- Non-expert XAI: Developing interfaces that provide "semantic transformations" of technical model outputs into terms a domain expert understands.
- Automated Monitoring: Creating XAI tools that can automatically trigger alerts or suggest model adjustments when performance degrades.
- Computational Efficiency: Researching "proxy models" or faster approximation strategies to make XAI feasible for high-speed industrial applications.
Summary updated