Towards Trustworthy Autonomous Systems via Conversations and Explanations

Balint GyevnarAAAI 2024 · DOI

Introduction

As autonomous systems like self-driving cars transition from controlled testing environments to public roads, they encounter a significant "trust gap." Most modern AI relies on deep learning models that function as black boxes, making high-stakes decisions without the ability to explain their reasoning. Traditional Explainable AI (XAI) methods, such as saliency maps or feature weights, are often too technical for laypeople and struggle to account for the complexities of multi-agent environments.

This research introduces CEMA (Causal Explanations in Multi-Agent systems), a framework designed to bridge this gap through human-like dialogue. CEMA aims to restore human agency by providing causal, contrastive explanations that align with human mental models, moving beyond technical visualizations toward social exchange.

Methodology: The CEMA Framework

The core of the CEMA framework is a "roll-back and simulate" approach to causal selection. Rather than relying on a fixed mathematical structure, CEMA leverages the agent's internal predictive models to discover causes dynamically through simulation.

The Counterfactual Causal Selection process within the CEMA framework.

As illustrated in the figure above, the process involves several key stages:

  1. Rollback: Upon receiving a query via the Human-Agent Interface, the system "rewinds" the environment's state to the moment preceding the decision.
  2. Counterfactual Sampling: A probabilistic model generates various "what-if" scenarios (counterfactual worlds) to explore alternative outcomes.
  3. Causal Selection: Using a Counterfactual Effect Size Model, the system identifies correlations across these simulations. A feature is assigned a high causal effect size if a specific outcome (e.g., braking) consistently occurs only when that feature (e.g., a pedestrian) is present.
  4. Ranking and Attribution: The system ranks these features to isolate the most relevant causes, effectively filtering out "noise" from irrelevant distractor agents in the environment.

The ultimate goal is to integrate this causal engine with a Natural Language Processing (NLP) pipeline to facilitate personalized, conversational responses that adapt to the user's mental model.

Evaluation and Results

The CEMA framework was evaluated using complex autonomous driving motion-planning scenarios involving multiple interacting agents. A user study with $N=200$ participants compared CEMA’s generated explanations against human-elicited explanations and other baselines.

Key findings include:

  • Human-Level Performance: Participants ranked CEMA’s explanations as being at least as helpful as those written by humans.
  • Trust Calibration: The study demonstrated a measurable positive effect on participant trust in the autonomous system following CEMA's explanations.
  • Robustness: The framework successfully identified correct causes even in crowded environments with numerous causally irrelevant agents.

Discussion and Future Directions

CEMA shifts the XAI paradigm from "opening the box" to "social explanation." By mimicking the human cognitive process of contrasting actual events with counterfactuals, it provides more intuitive insights. Furthermore, by using the agent's own internal models for simulation, CEMA avoids the pitfalls of trying to map complex neural networks to rigid, high-level Bayesian structures.

However, several challenges remain. While the causal selection engine is functional, full integration with a conversational agent that maintains a "mental model" of the user is still in development. Future work will focus on bridging the gap between causal rankings and natural language fluency, ensuring the system avoids over-explaining obvious facts. Additionally, the researchers plan to test CEMA’s versatility in other sequential decision-making domains beyond autonomous driving.

Summary updated

BibTeX
@inproceedings{gyevnar2024towards,
  author = {Balint Gyevnar},
  title = {{Towards Trustworthy Autonomous Systems via Conversations and Explanations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30395},
}

Loading related content...

Figure