Conversational and Interactive Interpretability for Machine Learning

424 concepts246 papers2 proposals

When we ask a human expert to explain a decision, we rarely accept a single, unchangeable chart as the final answer. If a doctor diagnoses a patient, we ask follow-up questions: "Why did you prioritize this symptom over that one?" or "What would change if the patient didn't have a fever?" Human explanation is fundamentally an iterative, social, and dialectical process.

Yet, traditional Explainable AI (XAI) treats interpretability as a static deliverable. Systems output a single "one-shot" report—such as a feature importance list or a saliency heatmap—and consider the job done. As Watson argues, this static paradigm fails because true comprehension is inherently process-oriented. To bridge the gap between human intuition and complex models, interpretability must shift from passive data dumps to interactive, conversational dialogues.

Why Static Explanations Fall Short

Static explanations provide a rigid, unidirectional view of a model's internal state. While a heatmap might show which pixels a computer vision model attended to, it remains inherently passive. Users are left unable to interactively test how a model's output would change under specific, semantically meaningful transformations (Suresh et al.). This passivity creates several critical points of failure:

  1. The Illusion of Correctness: When users passively consume static outputs, they struggle to identify when a model's reasoning is flawed. Fok demonstrates that static explanations frequently fail to prevent humans from accepting plausible-sounding but incorrect algorithmic reasoning. Without a way to verify the logic, explanations become mere stories.
  2. Failure in High-Stakes Contexts: In specialized domains like medicine or law, professional users have context-dependent questions that a single visualization cannot anticipate (Kim et al.). For example, static counterfactual explanations often fail to alter the decisions of experts because they cannot probe specific model sensitivities on demand (Yacoby et al.).
  3. Cognitive Mismatch: Human psychology relies on "folk concepts" of behavior to understand complex actions. We resolve contradictions by testing assumptions against contrast cases over time. One-shot, non-interactive explanations do not allow users to map the boundaries of a system's decision-making logic or resolve contradictions in real-time (Jacovi et al.).

The Mechanism: Explanations as Dialectical Argumentation

To resolve these limitations, interactive interpretability frameworks restructure the explanation process as a live dialogue. Instead of treating an explanation as a raw data transfer, the system and the user engage in a structured back-and-forth exchange.

As Mittelstadt et al. propose, this can be modeled using dialectical argumentation. The interaction is framed as a game or dialogue where the system provides evidence for its predictions, and the user can provide counter-evidence, challenge assumptions, or ask follow-up questions.

   [ User Input / Query ]  ──(What if feature X changes?)──►  [ Interactive Engine ]
                                                                     │
                                                              Causal / Symbolic
                                                                 Evaluation
                                                                     ▼
   [ User Mental Model ]   ◄───(Attacks / Supports Rules)─────  [ Argumentation ]

This conversational mechanism often relies on formal argumentation theory, mapping feature interactions as "attacks" or "supports" within a structured framework (Rago et al.). For instance, by structuring an AI's agency around causal action influence models and dialogue protocols, an agent can intuitively answer not just why it took an action, but why not an alternative path (Madumal).

In high-stakes environments, this dialectic reasoning—such as explicitly presenting arguments both for and against a specific outcome—gives human experts the necessary friction to deeply evaluate recommendations rather than blindly trusting them (Strong et al.).

How Interactive Probing Works

In practice, moving from static reports to dynamic conversations requires technical mechanisms that allow users to manipulate, query, and steer models on the fly. Researchers have developed several distinct approaches to enable this interactivity:

Query Languages and Algebraic Operators

Rather than relying on pre-scripted templates, frameworks like i-Algebra introduce a library of composable algebraic operators and SQL-like query languages (Pang et al.). This allows users to perform complex "drill-down" and "what-if" analyses across different layers of a deep neural network, transforming model debugging into an active interrogation.

Natural Language and Generative Interfaces

With the rise of Large Language Models, abstract explainability data can be translated into natural language chatbots. Systems can fine-tune LLMs to act as domain-specific conversational partners, allowing non-experts to query model behavior using free-form text (Bokstaller et al.).

In medical imaging, this manifests as generative visual question answering (VQA). Instead of viewing a static diagnostic label, a pathologist can interactively ask questions about specific localized patches within a gigapixel whole slide image (Zhu et al.).

Input Editing and Counterfactual Transplants

Another powerful mechanism is direct semantic manipulation. Tools like CardioCNN allow users to perform "counterfactual image transplants"—such as moving patches of healthy tissue onto a diseased X-ray—to observe exactly how and when the model's prediction flips (Lane et al.). This active steering shifts the user from a passive consumer to an active prober of the model's boundaries (Parra).

Building a Shared Mental Model

Ultimately, the goal of conversational interpretability is to facilitate a collaborative dialogue where both parties build and refine a shared mental model (Suthers et al.).

When communication is bidirectional, the explanation process does not just benefit the human; it can also reinforce the system itself. For example, requiring a model to articulate its internal concepts allows users to identify underlying misconceptions and provide real-time feedback to update the system's symbolic logic or prompt memory (Madaan et al., Gilpin).

By forcing users to actively engage with evidence, contest arguments, and sequentially refine their queries, interactive interpretability transforms machine learning models from inscrutable black boxes into accountable, conversational partners.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_static_explanation_limitation_449f86ca,
  title = {static explanation limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-explanation-limitation/rdr_449f86cae1b24d11a010a82c700b5a07},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_static_explanation_limitation_449f86ca_v1,
  title = {static explanation limitation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/static-explanation-limitation/rdr_449f86cae1b24d11a010a82c700b5a07/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake