Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach

Masahiro Kaneko, Graham Neubig, Naoaki Okazaki

Abstract

Traditional explainable AI (XAI) often relies on one-way justifications that users may find difficult to interpret or correct. This paper introduces a novel framework for human-system collaboration where models engage in multi-turn discussions to solve complex Natural Language Inference (NLI) tasks. By leveraging a new dataset of human-human dialogues and "few-shot-discussion" prompting, the researchers achieved accuracy gains of up to 25 points on adversarial benchmarks.

Introduction: Beyond the Monologue

Most current XAI systems operate as a "monologue": the model provides a prediction and a technical justification, leaving the human with a binary choice to accept or reject. This approach suffers from an expertise gap—where non-experts cannot interpret technical heatmaps—and a lack of flexibility for humans to provide corrective feedback.

The authors propose that human problem-solving is inherently collaborative and iterative. By shifting from static explanations to dynamic discussions, AI systems can better resolve lexical ambiguities and refine their logic through natural interaction.

The Discussion-Based Framework

The core of this research is a framework designed for Natural Language Inference (NLI), where the system must determine if a premise entails, contradicts, or is neutral toward a hypothesis.

Figure 1: An example of a human-system discussion resolving gender ambiguity in an NLI task.

The researchers implemented three primary prompting strategies:

  1. Zero-shot: Standard task instructions without examples.
  2. Few-shot: Standard examples of premises, hypotheses, and labels.
  3. Few-shot-discussion: Full dialogue examples that demonstrate how a system should respond to human objections and refine its reasoning.

To scale this approach, the team generated "pseudo-discussions" using large language models (LLMs) like GPT-3.5 to fine-tune smaller open-source models such as MPT-7B and Falcon-7B, proving that synthetic dialogue data is a viable training resource.

Experimental Setup and Datasets

The study utilized the Stanford Natural Language Inference (SNLI) dataset to identify ambiguous cases and the Adversarial NLI (ANLI) benchmark to test robustness. A key contribution is a new Human-Human Discussion Dataset, consisting of transcribed dialogues where annotators debated difficult NLI problems. These real-world interactions served as the gold standard for teaching the system argumentative capabilities.

Key Results and Performance

The transition to a discussion-based model yielded significant performance improvements, particularly on challenging adversarial tasks designed to fool standard models.

Figure 4: Accuracy improvements comparing standard few-shot prompting to the few-shot-discussion approach.

As shown in the results, the few-shot-discussion approach improved accuracy by up to 25 points on the ANLI task compared to standard baselines. Furthermore, the system's reasoning quality—measured via BERTScore—more closely aligned with human logic, indicating that the model wasn't just guessing but actually "thinking out loud" more effectively.

Insights: Overcoming the Compliance Problem

A significant hurdle in human-AI interaction is "compliance": LLMs often agree with a human's incorrect argument if presented confidently. This research demonstrates that training on discussion examples helps models "stand their ground" when they are correct while remaining open to valid human corrections. This balance is crucial for high-stakes domains like legal or medical reasoning, where accuracy is paramount.

Conclusion and Future Directions

While discussion-based prompts increase computational costs due to their length, the massive accuracy gains suggest a viable path for more reliable AI. Future work will focus on expanding the diversity of human dialogue data and applying this framework to complex reasoning tasks beyond NLI, ensuring that AI systems can act as true collaborative partners rather than just black-box predictors.

Summary updated

Abstract

Humans work together to solve common problems by having discussions, explaining, and agreeing or disagreeing with each other. Similarly, if a system can have discussions with humans when solving tasks, it can improve the system's performance and reliability. In previous research on explainability, it has only been possible for the system to make predictions and for humans to ask questions about them rather than having a mutual exchange of opinions. This research aims to create a dataset and computational framework for systems that discuss and refine their predictions through dialogue. Through experiments, we show that the proposed system can have beneficial discussions with humans improving the accuracy by up to 25 points in the natural language inference task.
BibTeX
@inproceedings{kaneko2024solving,
  author = {Masahiro Kaneko and Graham Neubig and Naoaki Okazaki},
  title = {{Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach}},
  booktitle = {EACL (Findings)},
  year = {2024},
  url = {https://openreview.net/forum?id=QZxvnPGrBx},
}
Figure