Reconciling Explanations in Multi-Model Systems through Probabilistic Argumentation

Shengxin Hong, Xiuyi Fan

Abstract

Explainable Artificial Intelligence (XAI) has become critical in enhancing the transparency and trustworthiness of AI systems, especially as these systems are increasingly deployed in high-stakes domains such as healthcare and finance. Despite the progress made in developing explanation generation techniques for individual machine learning (ML) models, significant challenges remain in achieving coherent and comprehensive explanations in multi-model systems. This paper addresses these challenges by focusing on the explanation reconciliation problem (ERP) within multi-model systems. Traditional explanation generation techniques often fall short in multi-model systems contexts, where explanations from different models can conflict and fail to form a cohesive narrative. Through the use of probabilistic argumentation and knowledge representation techniques, we propose a framework for generating holistic explanations that align with human cognitive processes. Our approach involves mapping uncertain explanation information to probabilistic arguments and introducing criteria for explanation reconciliation based on user perspectives such as optimism, pessimism, and fairness. In addition, we introduce the relative independence assumption to optimize the search space for computational explanations.

Introduction: Beyond the Single-Model "Black Box"

While Explainable AI (XAI) has made great strides in helping us understand why a single machine learning model makes a specific prediction, real-world AI systems are rarely that simple. In fields like healthcare or finance, decisions are often the result of a complex pipeline of multiple models feeding into one another. When these models provide conflicting information, creating a single, coherent explanation becomes a significant challenge.

The Explanation Reconciliation Problem (ERP)

Most XAI techniques, such as LIME or SHAP, are "local"—they identify which features influenced a specific model's output. However, modern AI systems often consist of chains of models where the output of one serves as the input for another. This architecture introduces two major hurdles:

  1. Aggregation: Weaving individual local explanations into a cohesive story that reflects the behavior of the entire system.
  2. Conflict: Different models may operate in different probability spaces or have access to incomplete information, leading to "disagreements." For instance, an MRI analysis model might suggest a high risk of disease, while a cognitive test model suggests a low risk.

The authors define this as the Explanation Reconciliation Problem (ERP): the necessity of finding a consistent probability distribution across the entire system that is interpretable by a human user.

A multi-model system architecture where outputs from image analysis (a) and cognitive tests (b) feed into risk assessment (c) and diagnostic (d) models.

Methodology: Probabilistic Argumentation

To address the ERP, the paper proposes a framework rooted in Probabilistic Argumentation. The core components include:

  • Probabilistic Rules (p-rules): The system maps model dependencies into formal rules (e.g., "If Model A predicts X, then Model B predicts Y with probability $P$").
  • Multi-Model System Graph: The system is represented as a directed graph where nodes are models and edges represent data flows.
  • Reconciliation Criteria: Recognizing that humans interpret uncertainty differently, the framework offers three "perspectives" to resolve conflicts:
  • Optimistic Criterion: Maximizes the probability of a desired outcome (radical reasoning).
  • Pessimistic Criterion: Minimizes the probability of a desired outcome (conservative reasoning).
  • Laplace Criterion: Uses the principle of maximum entropy to distribute probabilities fairly when information is missing.
  • Relative Independence Assumption (RIA): To mitigate the computational complexity of calculating joint probabilities (an NP-hard problem), the authors introduce RIA. This assumes that models are independent if one cannot "reach" the other in the system graph, significantly optimizing the search space.

Clinical Case Study

The framework's utility is demonstrated through a clinical diagnostic case study involving four models: MRI Image Analysis (Model A), Cognitive Test Analysis (Model B), Brain Risk Assessment (Model C), and Final Diagnosis (Model D). This example illustrates how the framework handles conflicting inputs—such as a "sick" MRI versus a "healthy" cognitive test—to reach a reconciled diagnostic probability.

The probabilistic representation of the clinical diagnostic case study, showing the dependencies and weights between different model outputs.

Key Insights and Results

The authors highlight several critical advancements:

  • From Local to Holistic: Explaining the relationship between models is as vital as explaining the models themselves.
  • Human-Centric Uncertainty: By providing multiple reconciliation criteria, the framework acknowledges that a "correct" explanation often depends on the user's risk tolerance or professional role.
  • Computational Efficiency: The RIA allows the system to ignore unrelated factors, preventing the exponential complexity typically associated with probabilistic logic.

Qualitative results from the case study showed that using the Laplace Criterion provided a neutral, fair assessment (e.g., a 0.07 probability of early Alzheimer's despite conflicting data). Conversely, the Optimistic Criterion could highlight a higher probability (0.28) for specific conditions, useful for exploring worst-case scenarios.

Conclusion and Future Work

While the framework provides a robust foundation for multi-model XAI, challenges remain. Future work will focus on scaling the system for massive industrial pipelines and conducting empirical user studies to confirm which reconciliation perspectives best align with human decision-making in high-stakes environments.

Summary updated

Abstract

Explainable Artificial Intelligence (XAI) has become critical in enhancing the transparency and trustworthiness of AI systems, especially as these systems are increasingly deployed in high-stakes domains such as healthcare and finance. Despite the progress made in developing explanation generation techniques for individual machine learning (ML) models, significant challenges remain in achieving coherent and comprehensive explanations in multi-model systems. This paper addresses these challenges by focusing on the explanation reconciliation problem (ERP) within multi-model systems. Traditional explanation generation technique often fall short in multi-model systems contexts, where explanations from different models can conflict and fail to form a cohesive narrative. Through the use of probabilistic argumentation and knowledge representation techniques, we propose a framework for generating holistic explanations that align with human cognitive processes. Our approach involves mapping uncertain explanation information to probabilistic arguments and introducing criteria for explanation reconciliation based on user perspectives such as optimism, pessimism, fairness. In addition, we introduce the relative independence assumption to optimise the search space for computational explanations.
BibTeX
@misc{hong2024reconciling,
  author = {Shengxin Hong and Xiuyi Fan},
  title = {{Reconciling Explanations in Multi-Model Systems through Probabilistic Argumentation}},
  year = {2024},
  eprint = {2404.13419},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.13419},
}
Figure