Introduction
As artificial intelligence (AI) transitions from experimental laboratories to the core of industrial operations, the "black-box" nature of modern machine learning (ML) models has emerged as a significant barrier to deployment. While the academic community has responded with a surge in Explainable AI (XAI) research, a disconnect remains between theoretical advancements and the practical requirements of the producing industries. This paper investigates the practical relevance of XAI across the ML life cycle, contrasting industrial reality with the current state of academic research.
Methodology
To bridge the gap between theory and practice, a qualitative study was conducted featuring 36 semi-structured interviews with a diverse array of stakeholders. Participants included data scientists, ML researchers, managers, safety engineers, and service technicians from organizations such as Siemens AG, various startups, and technical associations.
The study utilized the CRISP-ML (Cross-Industry Standard Process for Machine Learning) framework to map XAI needs across six distinct stages:
- Business and Data Understanding
- Data Preparation
- Modelling
- Evaluation
- Deployment
- Monitoring and Maintenance
By comparing interview insights with a comprehensive literature review, the researchers identified critical alignments and mismatches between industrial needs and academic focus.
Bridging the Gap: Industrial Needs vs. Academic Focus
The "Expert Trap"
A primary finding is the existence of an "expert trap." Most contemporary XAI tools, such as SHAP or LIME, are designed by data scientists for data scientists. However, in industrial settings, the primary consumers of explanations are often domain experts—such as mechanical engineers or maintenance technicians—who possess deep contextual knowledge but lack specialized ML expertise. There is a pressing need for XAI systems to communicate in the "language of the domain" rather than in abstract statistical metrics.
XAI as a Competitive Advantage
In the industrial sector, explainability is evolving from a technical requirement into a strategic differentiator. As customers become more AI-mature, they increasingly demand transparency. Companies that can provide interpretable and trustworthy AI systems gain a significant competitive edge, particularly in high-stakes environments where safety and reliability are paramount.
The Neglected Monitoring Phase
Academic research is heavily weighted toward the "Evaluation" phase—explaining model behavior post-training but pre-deployment. Conversely, industrial practitioners are deeply concerned with the "Monitoring and Maintenance" phase. When a model’s performance drifts in a factory setting, technicians require XAI to perform rapid root-cause analysis—distinguishing, for example, between a hardware sensor failure and a fundamental shift in data distribution.
Key Findings Across the ML Life Cycle
- Data Preparation: While academia focuses on model outputs, industry sees immense value in using XAI to detect data biases and incorrect labels during the collection phase. Tools like influence functions are theoretically available but have yet to see widespread industrial adoption.
- Modelling: There is a growing desire for "XAI-guided" development. Rather than using XAI solely for post-hoc analysis, practitioners want to use explanations to inform hyperparameter tuning and architecture selection.
- Regulatory Compliance: The impending EU AI Act is a major catalyst for XAI adoption. High-risk applications will likely require transparency by law, making XAI a mandatory component of the industrial certification process.
- Computational Scalability: Many popular XAI methods are computationally intensive. In real-time industrial applications, the high latency associated with calculating complex explanations (e.g., exact Shapley values) remains a significant technical bottleneck.
Conclusion and Future Directions
The study confirms that while XAI is highly valued in industry, current academic tools are often misaligned with the needs of non-expert users and the operational realities of the production environment. To close this gap, future research should prioritize:
- Human-Centric Design: Developing interfaces that provide semantic transformations of technical outputs into domain-specific insights.
- Automated Monitoring: Integrating XAI into automated alerting systems to facilitate real-time troubleshooting in production.
- Efficiency: Optimizing XAI algorithms or utilizing proxy models to ensure explainability does not compromise system performance.
Ultimately, the successful integration of AI in industry depends not just on the accuracy of the models, but on the ability of humans to understand, trust, and manage them throughout their entire life cycle.
Summary updated