The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research

Thomas Decker, Ralf Gross, Alexander Koebler, Michael Lebacher, Ronald Schnitzer, Stefan H. Weber

TL;DR

This paper bridges the gap between academic Explainable AI (XAI) research and its industrial application by mapping XAI needs across the entire Machine Learning life cycle. Through 36 stakeholder interviews, the authors reveal that while research is heavily focused on model evaluation for data scientists, industrial reality demands XAI tools for data preparation, monitoring, and non-expert users.

What Problem Does This Paper Address?

Despite the explosion of AI in industry, moving from a "Proof of Concept" to a deployed, reliable system remains a significant hurdle. A primary culprit is the "black-box" nature of modern models, which hampers trust, complicates debugging, and makes regulatory compliance difficult.

The authors argue that current XAI research is too narrow. Most academic papers focus on the "Evaluation" phase—explaining why a model made a specific prediction after it has already been trained. However, industrial practitioners need "explainability" at every step: from understanding data biases during collection to performing root-cause analysis when a model's performance drifts in production. There is a fundamental mismatch between the tools academia is building and the practical needs of the "producing industries" (e.g., manufacturing and automation).

Methods

The researchers conducted a qualitative study consisting of 36 semi-structured interviews with a diverse range of roles, including data scientists, ML researchers, managers, safety engineers, and service technicians from Siemens AG, startups, and technical associations.

To organize their findings, they used the CRISP-ML (Cross-Industry Standard Process for Machine Learning) framework, which divides the ML life cycle into six stages:

  1. Business and Data Understanding
  2. Data Preparation
  3. Modelling
  4. Evaluation
  5. Deployment
  6. Monitoring and Maintenance

By mapping interview insights against current academic literature for each stage, they identified "alignments" (where research meets needs) and "mismatches" (where industry is left wanting).

Datasets

As this is a qualitative research paper focused on industrial practice and literature review, it does not introduce or use standard ML training datasets (like ImageNet or GLUE). The primary "data" consists of the transcribed protocols from the 36 expert interviews and a survey of existing XAI literature.

Key Insights and Ideas

  • XAI as a Competitive Advantage: In the industrial market, explainability is no longer just a "nice-to-have." It is becoming a "distinguishing factor" that companies use to build trust with customers who are increasingly AI-mature and skeptical of opaque systems.
  • The "Expert" Trap: Most XAI tools (like SHAP or LIME) are designed by data scientists for data scientists. In reality, the people who need explanations are often domain experts (e.g., a mechanical engineer) or technicians who lack deep ML knowledge. The paper highlights a desperate need for XAI to speak the "language of the domain."
  • The Neglected "Monitoring" Phase: While academia loves the "Evaluation" phase, industry is deeply concerned with "Monitoring and Maintenance." When a model fails in a factory, technicians need XAI to tell them why it failed and how to fix it (e.g., "the sensor is dusty" vs. "the data distribution has shifted").
  • Scalability is a Dealbreaker: Many popular XAI methods are computationally expensive. The authors found that the high cost of calculating explanations (like Shapley values) often prevents their use in real-time industrial deployment.

Key Results

The paper provides a comprehensive "map" (Figure 2) of XAI across the ML life cycle. Key qualitative findings include:

  • Data Collection: XAI is being used to detect "wrong labels" and data bias, though academic tools for this (like influence functions) are not yet popular in practice.
  • Modelling: There is a strong desire for "XAI-guided" development, such as using explanations to choose hyperparameters or network architectures, rather than just evaluating the final result.
  • Regulation: The upcoming EU AI Act is a major driver. High-risk AI applications will likely require transparency and explainability by law, making XAI a mandatory component of the certification process.
  • Human-Centricity: The study confirms that "Visual Analytics" and human-computer interaction (HCI) are just as important as the underlying algorithms. An explanation is useless if the end-user can't interpret it.

Limitations and Future Work

The authors acknowledge that their sample size (36 interviews) is relatively small and focused primarily on the "producing industries," meaning the results might differ for sectors like finance or healthcare.

Future Work should focus on:

  1. Non-expert XAI: Developing interfaces that provide "semantic transformations" of technical model outputs into terms a domain expert understands.
  2. Automated Monitoring: Creating XAI tools that can automatically trigger alerts or suggest model adjustments when performance degrades.
  3. Computational Efficiency: Researching "proxy models" or faster approximation strategies to make XAI feasible for high-speed industrial applications.

Summary updated

Abstract

In this paper, we investigate the practical relevance of explainable artificial intelligence (XAI) with a special focus on the producing industries and relate them to the current state of academic XAI research. Our findings are based on an extensive series of interviews regarding the role and applicability of XAI along the Machine Learning (ML) lifecycle in current industrial practice and its expected relevance in the future. The interviews were conducted among a great variety of roles and key stakeholders from different industry sectors. On top of that, we outline the state of XAI research by providing a concise review of the relevant literature. This enables us to provide an encompassing overview covering the opinions of the surveyed persons as well as the current state of academic research. By comparing our interview results with the current research approaches we reveal several discrepancies. While a multitude of different XAI approaches exists, most of them are centered around the model evaluation phase and data scientists. Their versatile capabilities for other stages are currently either not sufficiently explored or not popular among practitioners. In line with existing work, our findings also confirm that more efforts are needed to enable also non-expert users’ interpretation and understanding of opaque AI models with existing methods and frameworks.
BibTeX
@inproceedings{decker2023thousand,
  author = {Thomas Decker and Ralf Gross and Alexander Koebler and Michael Lebacher and Ronald Schnitzer and Stefan H. Weber},
  title = {{The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research}},
  booktitle = {HCI},
  year = {2023},
  url = {https://openreview.net/forum?id=gpIkTRIJ8h2},
}

Loading related content...

Figure