Domain-Level Explainability - A Challenge for Creating Trust in Superhuman AI Strategies

Jonas Andrulis, Ole Meyer, Grégory Schott, Samuel Weinbach, Volker GruhnarXiv 2020 · OpenReview

Abstract

For strategic problems, intelligent systems based on Deep Reinforcement Learning (DRL) have demonstrated an impressive ability to learn advanced solutions that can go far beyond human capabilities, especially when dealing with complex scenarios. While this creates new opportunities for the development of intelligent assistance systems with groundbreaking functionalities, applying this technology to real-world problems carries significant risks and therefore requires trust in their transparency and reliability. With superhuman strategies being non-intuitive and complex by definition and real-world scenarios prohibiting a reliable performance evaluation, the key components for trust in these systems are difficult to achieve. Explainable AI (XAI) has successfully increased transparency for modern AI systems through a variety of measures, however, XAI research has not yet provided approaches enabling domain level insights for expert users in strategic situations. In this paper, we discuss the existence of superhuman DRL-based strategies, their properties, the requirements and challenges for transforming them into real-world environments, and the implications for trust through explainability as a key technology.

Introduction: The Trust Gap in Superhuman AI

Deep Reinforcement Learning (DRL) has achieved remarkable success in mastering complex strategic games, often surpassing human expertise. However, as we transition these technologies into critical real-world domains like logistics, finance, and defense, a significant "trust gap" emerges. Human experts often find superhuman strategies counter-intuitive or even nonsensical, hindering the adoption of these powerful systems. This paper addresses the critical bottleneck in DRL deployment: the disconnect between superhuman performance and human understanding.

The Paradox of Superhuman Performance

DRL agents are uniquely capable of finding optimal solutions in dynamic and hostile environments without requiring human-labeled data. This autonomy allows them to discover "Move 37" moments—strategies that violate established domain wisdom but lead to victory. The authors compare these strategies to "disruptive innovations" in business; just as a new business model can render old theories obsolete, a DRL agent can find winning moves that contradict human expertise.

While celebrated in games like Go or Dota 2, such opaque brilliance is a liability in high-stakes professional environments. A strategy that cannot be explained is often indistinguishable from a system failure to a human commander or executive. Current XAI methods, such as heatmaps showing neural network focus at a pixel level, are too low-level for strategic decision-making.

Frameworks for Strategic Complexity and Trust

To bridge this gap, the authors propose a taxonomy that categorizes the inherent difficulties in explaining DRL-based strategies. This taxonomy identifies seven dimensions of strategic complexity (C1–C7), ranging from spatial and temporal reasoning to the management of massive action spaces.

Taxonomy of Strategic Complexity (C1-C7) and Strategic Explainability requirements (E1-E4).

Understanding these complexities is the first step toward developing "Strategic Explainability" (E1–E4). These requirements focus on high-level concepts like future projections and hypothetical scenarios rather than technical metrics.

Trust in an AI system is not solely dependent on its performance. The authors introduce a conceptual model where trust is a function of both Robustness (the system's ability to perform consistently) and Explainability (the human's ability to understand the "why" behind an action).

The Trusted Zone: A conceptual model showing that trust requires both high robustness and high explainability for real-world delegation.

As illustrated in the "Trusted Zone" framework, a system that is highly robust but unexplainable remains outside the zone of acceptable delegation. Users cannot predict its behavior in novel situations or verify its logic against domain expertise, leading to potential rejection of even the most optimal strategies.

Moving Toward Domain-Level Explainability

The paper argues that for an AI to be trusted, it must communicate in the language of the domain. This requires shifting from technical XAI to "Strategic XAI." Key components include:

  • Future Projections (E1): Visualizing the agent's predicted sequence of events and competitor behaviors.
  • Hypothetical Scenarios (E2): Enabling "what-if" analysis to test the agent's response to different potential future states.
  • Risk and Uncertainty (E3/E4): Quantifying the model's confidence and its ability to detect out-of-distribution cases.
  • Contrastive Explanations: Explaining why the AI chose a specific strategy over a human-preferred alternative (e.g., "Why Strategy A instead of Strategy B?").

Conclusion

Achieving domain-level explainability is an immense technical challenge, but it is essential for the responsible deployment of superhuman AI. By shifting the focus from technical transparency to strategic communication, we can create systems that act not just as black-box optimizers, but as trusted partners for human experts in complex, real-world environments.

Summary updated

Abstract

For strategic problems, intelligent systems based on Deep Reinforcement Learning (DRL) have demonstrated an impressive ability to learn advanced solutions that can go far beyond human capabilities, especially when dealing with complex scenarios. While this creates new opportunities for the development of intelligent assistance systems with groundbreaking functionalities, applying this technology to real-world problems carries significant risks and therefore requires trust in their transparency and reliability. With superhuman strategies being non-intuitive and complex by definition and real-world scenarios prohibiting a reliable performance evaluation, the key components for trust in these systems are difficult to achieve. Explainable AI (XAI) has successfully increased transparency for modern AI systems through a variety of measures, however, XAI research has not yet provided approaches enabling domain level insights for expert users in strategic situations. In this paper, we discuss the existence of superhuman DRL-based strategies, their properties, the requirements and challenges for transforming them into real-world environments, and the implications for trust through explainability as a key technology.
BibTeX
@misc{andrulis2020domain,
  author = {Jonas Andrulis and Ole Meyer and Grégory Schott and Samuel Weinbach and Volker Gruhn},
  title = {{Domain-Level Explainability - A Challenge for Creating Trust in Superhuman AI Strategies}},
  year = {2020},
  eprint = {2011.06665},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.06665},
}
Figure