Abstract
Traditional explainable AI (XAI) often relies on one-way justifications that users may find difficult to interpret or correct. This paper introduces a novel framework for human-system collaboration where models engage in multi-turn discussions to solve complex Natural Language Inference (NLI) tasks. By leveraging a new dataset of human-human dialogues and "few-shot-discussion" prompting, the researchers achieved accuracy gains of up to 25 points on adversarial benchmarks.
Introduction: Beyond the Monologue
Most current XAI systems operate as a "monologue": the model provides a prediction and a technical justification, leaving the human with a binary choice to accept or reject. This approach suffers from an expertise gap—where non-experts cannot interpret technical heatmaps—and a lack of flexibility for humans to provide corrective feedback.
The authors propose that human problem-solving is inherently collaborative and iterative. By shifting from static explanations to dynamic discussions, AI systems can better resolve lexical ambiguities and refine their logic through natural interaction.
The Discussion-Based Framework
The core of this research is a framework designed for Natural Language Inference (NLI), where the system must determine if a premise entails, contradicts, or is neutral toward a hypothesis.
The researchers implemented three primary prompting strategies:
- Zero-shot: Standard task instructions without examples.
- Few-shot: Standard examples of premises, hypotheses, and labels.
- Few-shot-discussion: Full dialogue examples that demonstrate how a system should respond to human objections and refine its reasoning.
To scale this approach, the team generated "pseudo-discussions" using large language models (LLMs) like GPT-3.5 to fine-tune smaller open-source models such as MPT-7B and Falcon-7B, proving that synthetic dialogue data is a viable training resource.
Experimental Setup and Datasets
The study utilized the Stanford Natural Language Inference (SNLI) dataset to identify ambiguous cases and the Adversarial NLI (ANLI) benchmark to test robustness. A key contribution is a new Human-Human Discussion Dataset, consisting of transcribed dialogues where annotators debated difficult NLI problems. These real-world interactions served as the gold standard for teaching the system argumentative capabilities.
Key Results and Performance
The transition to a discussion-based model yielded significant performance improvements, particularly on challenging adversarial tasks designed to fool standard models.
As shown in the results, the few-shot-discussion approach improved accuracy by up to 25 points on the ANLI task compared to standard baselines. Furthermore, the system's reasoning quality—measured via BERTScore—more closely aligned with human logic, indicating that the model wasn't just guessing but actually "thinking out loud" more effectively.
Insights: Overcoming the Compliance Problem
A significant hurdle in human-AI interaction is "compliance": LLMs often agree with a human's incorrect argument if presented confidently. This research demonstrates that training on discussion examples helps models "stand their ground" when they are correct while remaining open to valid human corrections. This balance is crucial for high-stakes domains like legal or medical reasoning, where accuracy is paramount.
Conclusion and Future Directions
While discussion-based prompts increase computational costs due to their length, the massive accuracy gains suggest a viable path for more reliable AI. Future work will focus on expanding the diversity of human dialogue data and applying this framework to complex reasoning tasks beyond NLI, ensuring that AI systems can act as true collaborative partners rather than just black-box predictors.
Summary updated