Conversational XAI and Explanation Dialogues

Nils Feldhus2024 · DOI

This blog-style summary provides an overview of the position paper "Conversational XAI and Explanation Dialogues" by Nils Feldhus, presented at the 20th Annual Meeting of the Young Researchers’ Roundtable on Spoken Dialogue Systems (2024).

TL;DR

Current Explainable AI (XAI) often feels like a rigid Q&A session rather than a helpful conversation. This paper argues for a shift toward "human-centric" explanatory dialogues that use advanced dialogue management and insights from education (didactics) to make complex language models more transparent, trustworthy, and accessible to non-experts.

What Problem Does This Paper Address?

While Large Language Models (LLMs) are becoming ubiquitous, their inner workings remain a "black box" for most users. Existing XAI systems attempt to solve this by providing explanations, but they usually function in a "one-shot" manner: the user asks a question, and the system provides a static explanation.

Feldhus identifies a major gap: these systems lack contextual awareness and dialogue management. They don't account for the user's specific level of knowledge, nor do they allow for the natural back-and-forth—clarifications, arguments, or step-by-step instructions—that characterizes human learning. The challenge is to move from static explanations to dynamic, explanatory conversations that lower the barrier of entry for non-experts.

Methods

The paper outlines a research agenda focused on building a dedicated explanatory dialogue management system. Key components of this approach include:

  • Interdisciplinary Frameworks: Integrating theories from computational argumentation (how to structure a persuasive or logical point) and didactics (the science of teaching) to guide how a system should "instruct" a user.
  • LLM-Driven Interaction: Leveraging LLMs not just to generate the explanation itself, but to perform "dialogue state tracking"—essentially keeping track of what has been explained and what the user still needs to know.
  • Human-in-the-Loop Evaluation: Conducting user studies to compare LLM-generated explanations against traditional, template-based answers. The goal is to measure "likeability" and "perceived faithfulness" (whether the user thinks the explanation is honest).
  • Instructional Modeling: Investigating whether language models can detect if a "teacher" (or a tutoring bot) is following pedagogical best practices.

Datasets

While this position paper does not introduce a massive new dataset, it references several critical benchmarks and tools used in the author's ongoing work:

  • InterroLang & LLMCheckup: Tools for conversational examination of NLP models.
  • Teacher-Student Dialogue Corpora: Used to model instructional explanations and evaluate teaching strategies.
  • Test Suites for Faithfulness: Referenced (e.g., Atanasova et al., 2023) to analyze the gap between what a model says it's doing and what it is actually doing.

Key Insights and Ideas

  • Explanations as "Moves": Instead of seeing an explanation as a single block of text, the author views it as a series of "moves" in a dialogue—similar to how a teacher adjusts their strategy based on a student's confusion.
  • The Trust Hypothesis: The core belief is that users are more satisfied and trusting when they can "look behind the curtain" of an AI’s decision-making process through easy, conversational access.
  • The Faithfulness Gap: A crucial insight is the potential disconnect between perceived faithfulness (how convincing an explanation sounds) and actual faithfulness (how well the explanation reflects the model's internal logic). LLMs are very good at sounding convincing, even when they are wrong.
  • Tutoring as XAI: Framing XAI as a tutoring task allows researchers to use decades of educational research to improve how AI explains itself to humans.

Key Results

As a research interest paper, the focus is on preliminary findings and future directions:

  • LLM Versatility: LLMs are shown to be exceptionally good at synthesizing natural language explanations in various styles (concise vs. elaborate) and tracking the state of a conversation.
  • Annotation Challenges: Initial work on detecting "good teaching practices" reveals that it requires high-level expertise from human annotators to reach a consensus, suggesting that automating the evaluation of "good" explanations is a high-difficulty task.
  • Template vs. LLM: Ongoing studies suggest LLM-generated explanations may beat conventional templates in user preference, though their factual reliability remains a concern.

Limitations and Future Work

The author acknowledges several hurdles that the field must overcome:

  • Ground Truth Issues: LLMs often struggle with "hallucinations," making it difficult to establish a ground truth for what a "correct" explanation looks like.
  • Evaluation Metrics: There is a pressing need for better measures of "generated text quality," specifically regarding how well a bot sticks to a planned "lesson" or explanation path.
  • Societal Risks: Uncontrolled systems that "extrude falsehoods" pose risks; future work must focus on grounding explanations in actual data and model architecture to ensure accountability.
  • Personalization: A long-term goal is creating "personalized tutoring chatbots" that adapt to a user's specific personality, expertise, and mental model.

Summary updated

BibTeX
@misc{feldhus2024conversational,
  author = {Nils Feldhus},
  title = {{Conversational XAI and Explanation Dialogues}},
  year = {2024},
  doi = {10.18653/v1/2024.yrrsds-1.1},
}
Figure