Abstract
Current Explainable AI (XAI) systems often generate explanations in a vacuum, failing to account for the specific knowledge and beliefs of the human user. This paper proposes a formal, belief-based framework that integrates "Theory of Mind" (ToM) into the explanation process. By modeling the epistemic states and belief-revision mechanisms of both the explainer and the explainee, the framework enables AI agents to generate tailored, proactive, and context-aware explanations. We provide a formal characterization of subjective explanation and prove several theorems regarding the adequacy of an explainer's model of the user.
Introduction: The "Soliloquy" Problem in XAI
Explainable AI (XAI) has emerged as a critical field for building trust between humans and autonomous systems. However, a significant portion of current research treats explanation as a "soliloquy"—a one-way dump of internal reasoning or model weights. These systems often assume a shared knowledge base between the AI and the human, ignoring the fact that the two agents possess vastly different backgrounds, expertise, and current beliefs.
An explanation that is logically sound from the AI's perspective may be entirely incomprehensible to a user who lacks the necessary context. To bridge this gap, AI requires a "Theory of Mind" (ToM): the cognitive ability to attribute mental states—such as beliefs, desires, and intentions—to oneself and others. By incorporating ToM, AI can move beyond generic outputs toward personalized explanations that address the specific knowledge gaps of the user.
A Formal Framework for Epistemic States
The authors propose a theoretical framework that is "agent-type agnostic," meaning it can be applied to various AI architectures, from neural networks to symbolic logic systems. The core of this framework rests on three pillars:
- Epistemic States: The mental state of an agent (human or AI) is represented as an epistemic state. This abstraction allows the framework to model what an agent knows or believes without being tied to a specific physical implementation.
- Belief Operators: Using modal logic, the framework defines $B_i\phi$ to represent that agent $i$ believes proposition $\phi$. This allows the system to track nested beliefs (e.g., "The AI believes the user believes the system is safe").
- Belief Revision: Explanations are not just new facts; they often require the listener to change their mind. The framework uses revision operators $[\alpha]_i\phi$ to model how agent $i$ updates their beliefs when presented with information $\alpha$.
Defining Subjective Explanation
Traditional logic-based accounts of explanation often focus on objective truth. In contrast, this paper defines explanation subjectively. An explanation $\alpha$ is considered valid for an explainee (agent $j$) from the perspective of the explainer (agent $i$) if:
- Agent $i$ believes that $\alpha$ is true.
- Agent $i$ believes that providing $\alpha$ to agent $j$ will cause $j$ to revise their beliefs such that they understand the target concept.
- The explanation does not introduce logical contradictions within agent $j$’s revised mental state.
This shift acknowledges that the "best" explanation is not necessarily the most detailed one, but the one most effective at updating the user's specific mental model.
Theoretical Results
While the paper is primarily theoretical, it establishes several formal proofs that underpin the utility of ToM in XAI:
- Equivalence of States (Theorem 2): The authors prove that if two agents possess functionally equivalent epistemic states, an explanation that is objectively valid for one is guaranteed to be valid for the other. This provides a mathematical basis for grouping users by expertise level.
- The Adequacy Bound (Theorem 5): The quality of an explanation is strictly bounded by the "adequacy" of the explainer's model of the user. If the AI's Theory of Mind is inaccurate, it will inevitably fail to provide an optimal explanation, leading to either over-explanation (redundancy) or under-explanation (confusion).
- Social Dynamics: The framework successfully models complex scenarios, such as privacy-preserving explanations (where an explanation is tailored to be understood by one person but remain opaque to another) and multi-agent reasoning.
Proactive Explanation and Discrepancy
One of the most powerful applications of this framework is the ability to generate proactive explanations. In traditional systems, the user must ask "Why?". With ToM, an AI agent can simulate the user's mental state and identify "discrepancies"—instances where the user's beliefs diverge from the AI's knowledge in a way that might lead to error. The AI can then intervene with an explanation before the user even realizes there is a misunderstanding.
Conclusion and Future Work
This research provides the formal building blocks for a more human-centric approach to XAI. By treating explanation as a social, communicative act rooted in Theory of Mind, we can create AI that is more intuitive and effective.
However, significant challenges remain. The current framework assumes the AI has access to a model of the user's beliefs, which is difficult to achieve in practice. Future work must focus on how AI can actively learn and refine these user models through interaction, and how these formal logical structures can be integrated into the "black-box" explanations of deep learning models.
Summary updated