TL;DR
This paper tackles the challenge of making autonomous systems trustworthy by enabling them to explain their actions in a human-understandable way. They propose extending the Belief-Desire-Intention (BDI) agent model with "valuings" – essentially, preferences and affective responses – to generate explanations that align with how humans naturally explain their own behavior.
What Problem Does This Paper Address?
As autonomous systems become more integrated into our lives, building user trust is paramount. A key factor in fostering this trust is the system's ability to explain why it took a particular action. However, current explanation mechanisms often lack the nuance and familiarity that humans use when explaining themselves. This paper addresses this gap by proposing a formal framework for generating explanations that leverage concepts familiar to humans, specifically drawing from social science research on folk psychology. The goal is to make explanations comprehensible and useful, allowing users to better understand, coordinate with, and anticipate the behavior of autonomous agents.
Methods
The authors propose a formal framework for generating explanations based on a BDI agent model augmented with "valuings." This framework utilizes a goal tree structure, where nodes represent goals and actions, and are decomposed using AND, SEQ, or OR logic. Valuings are incorporated as preferences over options, influenced by aspects like quality, cost, and distance.
The core of their contribution is an explanation function, denoted as E, which takes a question ("why did you do X?") and an agent's execution trace to produce explanatory factors. These factors include:
- Beliefs: Conditions that had to hold for an action to be performed.
- Valuings: Preferences for certain outcomes over others (e.g., preferring good coffee to bad coffee).
- Desires (Motivations): The underlying goals driving the action.
- Preparatory Factors: Actions taken to enable future actions.
- Failure Handling: Explanations for why an action failed and an alternative was chosen.
They also describe an algorithm for rendering these explanatory factors into natural language, aiming for conciseness and clarity.
Datasets
Not discussed in the paper. The paper focuses on a running example of an agent deciding how to get coffee and uses this to illustrate its framework and evaluation.
Key Insights and Ideas
- Human-like Explanations: The central idea is that by mirroring human explanatory constructs (beliefs, desires, and now "valuings"), autonomous systems can generate explanations that are more intuitive and comprehensible to users.
- The Power of "Valuings": The paper highlights "valuings" as a distinct and crucial concept for explaining behavior, representing affective responses and preferences that go beyond mere beliefs or goals. This is a key extension to traditional BDI models for explanation.
- Contextual Explanations: The framework allows for explanations that are sensitive to the specific context, including factors like available money, location, and even the failure of prior attempts.
- Bridging the Gap: By formalizing these concepts, the paper provides a computational mechanism for generating explanations, moving beyond theoretical discussions to practical implementation.
Key Results
The paper presents two human participant evaluations to assess the comprehensibility and usability of their generated explanations.
- Evaluation 1: Participants ranked explanations that combined valuings and beliefs (E1) and concrete explanations with specific details (E3) as better than those relying solely on one factor or abstract statements. This suggests that a combination of factors, particularly including valuings, is effective.
- Evaluation 2: This study reinforced the findings, showing that explanations incorporating valuing, belief, and desire components were preferred. The presence of valuing components showed the strongest correlation with higher preference. Explanations that included "links" (preparatory factors) were not preferred.
While specific quantitative metrics like accuracy are not detailed in this extended abstract, the qualitative results strongly support the effectiveness of their approach.
Limitations and Future Work
- Complexity of Explanations: While the authors aim for conciseness, the generated explanations can still be somewhat complex, especially when dealing with multiple factors and failures. The paper mentions filtering of explanatory factors as a direction for future work.
- Scope of "Valuings": The paper focuses on specific types of valuings (quality, cost, distance). Future work could explore a broader range of affective responses and preferences.
- Formal Definitions Omitted: As an extended abstract, formal definitions and detailed algorithmic descriptions are omitted, which would be crucial for full implementation and understanding.
- Further Evaluation: The authors suggest further work on evaluating the explanations, particularly in more complex scenarios and with different user groups. They also point to directions for future work in the closing section of their full AIJ paper.
Summary updated