The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research

Thomas Decker, Ralf Gross, Alexander Koebler, Michael Lebacher, Ronald Schnitzer, Stefan H. Weber

Introduction

As artificial intelligence (AI) transitions from experimental laboratories to the core of industrial operations, the "black-box" nature of modern machine learning (ML) models has emerged as a significant barrier to deployment. While the academic community has responded with a surge in Explainable AI (XAI) research, a disconnect remains between theoretical advancements and the practical requirements of the producing industries. This paper investigates the practical relevance of XAI across the ML life cycle, contrasting industrial reality with the current state of academic research.

Methodology

To bridge the gap between theory and practice, a qualitative study was conducted featuring 36 semi-structured interviews with a diverse array of stakeholders. Participants included data scientists, ML researchers, managers, safety engineers, and service technicians from organizations such as Siemens AG, various startups, and technical associations.

The study utilized the CRISP-ML (Cross-Industry Standard Process for Machine Learning) framework to map XAI needs across six distinct stages:

  1. Business and Data Understanding
  2. Data Preparation
  3. Modelling
  4. Evaluation
  5. Deployment
  6. Monitoring and Maintenance

By comparing interview insights with a comprehensive literature review, the researchers identified critical alignments and mismatches between industrial needs and academic focus.

Bridging the Gap: Industrial Needs vs. Academic Focus

The "Expert Trap"

A primary finding is the existence of an "expert trap." Most contemporary XAI tools, such as SHAP or LIME, are designed by data scientists for data scientists. However, in industrial settings, the primary consumers of explanations are often domain experts—such as mechanical engineers or maintenance technicians—who possess deep contextual knowledge but lack specialized ML expertise. There is a pressing need for XAI systems to communicate in the "language of the domain" rather than in abstract statistical metrics.

XAI as a Competitive Advantage

In the industrial sector, explainability is evolving from a technical requirement into a strategic differentiator. As customers become more AI-mature, they increasingly demand transparency. Companies that can provide interpretable and trustworthy AI systems gain a significant competitive edge, particularly in high-stakes environments where safety and reliability are paramount.

The Neglected Monitoring Phase

Academic research is heavily weighted toward the "Evaluation" phase—explaining model behavior post-training but pre-deployment. Conversely, industrial practitioners are deeply concerned with the "Monitoring and Maintenance" phase. When a model’s performance drifts in a factory setting, technicians require XAI to perform rapid root-cause analysis—distinguishing, for example, between a hardware sensor failure and a fundamental shift in data distribution.

Key Findings Across the ML Life Cycle

  • Data Preparation: While academia focuses on model outputs, industry sees immense value in using XAI to detect data biases and incorrect labels during the collection phase. Tools like influence functions are theoretically available but have yet to see widespread industrial adoption.
  • Modelling: There is a growing desire for "XAI-guided" development. Rather than using XAI solely for post-hoc analysis, practitioners want to use explanations to inform hyperparameter tuning and architecture selection.
  • Regulatory Compliance: The impending EU AI Act is a major catalyst for XAI adoption. High-risk applications will likely require transparency by law, making XAI a mandatory component of the industrial certification process.
  • Computational Scalability: Many popular XAI methods are computationally intensive. In real-time industrial applications, the high latency associated with calculating complex explanations (e.g., exact Shapley values) remains a significant technical bottleneck.

Conclusion and Future Directions

The study confirms that while XAI is highly valued in industry, current academic tools are often misaligned with the needs of non-expert users and the operational realities of the production environment. To close this gap, future research should prioritize:

  1. Human-Centric Design: Developing interfaces that provide semantic transformations of technical outputs into domain-specific insights.
  2. Automated Monitoring: Integrating XAI into automated alerting systems to facilitate real-time troubleshooting in production.
  3. Efficiency: Optimizing XAI algorithms or utilizing proxy models to ensure explainability does not compromise system performance.

Ultimately, the successful integration of AI in industry depends not just on the accuracy of the models, but on the ability of humans to understand, trust, and manage them throughout their entire life cycle.

Summary updated

Abstract

In this paper, we investigate the practical relevance of explainable artificial intelligence (XAI) with a special focus on the producing industries and relate them to the current state of academic XAI research. Our findings are based on an extensive series of interviews regarding the role and applicability of XAI along the Machine Learning (ML) lifecycle in current industrial practice and its expected relevance in the future. The interviews were conducted among a great variety of roles and key stakeholders from different industry sectors. On top of that, we outline the state of XAI research by providing a concise review of the relevant literature. This enables us to provide an encompassing overview covering the opinions of the surveyed persons as well as the current state of academic research. By comparing our interview results with the current research approaches we reveal several discrepancies. While a multitude of different XAI approaches exists, most of them are centered around the model evaluation phase and data scientists. Their versatile capabilities for other stages are currently either not sufficiently explored or not popular among practitioners. In line with existing work, our findings also confirm that more efforts are needed to enable also non-expert users' interpretation and understanding of opaque AI models with existing methods and frameworks.
BibTeX
@inproceedings{decker2023thousand,
  author = {Thomas Decker and Ralf Gross and Alexander Koebler and Michael Lebacher and Ronald Schnitzer and Stefan H. Weber},
  title = {{The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research}},
  booktitle = {HCI},
  year = {2023},
  url = {https://openreview.net/forum?id=gpIkTRIJ8h2},
}
Figure