Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach

Masahiro Kaneko, Graham Neubig, Naoaki Okazaki

Introduction

Deep learning models are often criticized as "black boxes" that provide one-way explanations. While Explainable AI (XAI) aims to justify model predictions, these methods typically require users to interpret complex internal states. This paper proposes a shift toward bidirectional collaboration, where systems and humans resolve ambiguities through multi-turn dialogue, mimicking human problem-solving. By allowing a system to not only explain its stance but also listen to and refine its logic based on human feedback, the researchers aim to achieve more accurate and robust outcomes in Natural Language Processing (NLP) tasks.

Methodology: Few-Shot-Discussion

The core of this approach is the "few-shot-discussion" framework. Unlike standard few-shot prompting—which provides static task examples—this method includes transcribed multi-turn discussions between humans debating the correct label. This teaches the Large Language Model (LLM) the logic of argumentation and how to adjust its internal reasoning based on external input.

The "few-shot-discussion" prompt structure, featuring a task description and a transcribed debate between two humans.

Key Components

  • Prompt Engineering: Prompts were designed to include a premise/hypothesis pair followed by a dialogue ("Human1" and "Human2") debating the inference label.
  • Model Selection: The study utilized state-of-the-art models including GPT-3.5 and ChatGPT.
  • Pseudo-Discussion Data: To scale training, the authors used LLMs to generate 2,000 synthetic "pseudo-discussions," which proved nearly as effective as manual human annotations.

Datasets and Evaluation

The study focused on Natural Language Inference (NLI), using the Stanford NLI (SNLI) and the more challenging Adversarial NLI (ANLI) datasets. The authors also curated a new dataset of oral discussions from 14 NLP-knowledgeable annotators. Evaluation went beyond standard accuracy, using BERTScore to compare system-generated arguments against human logic and manually assessing the system's ability to "object" to incorrect human reasoning.

Key Results

The discussion-based approach yielded significant performance improvements, particularly on difficult tasks. On specific NLI problems, the final label accuracy improved by up to 25 points after the discussion phase compared to the model's initial zero-shot prediction.

Comparative accuracy results showing the performance jump (up to 25 points) after implementing the discussion-based approach.
  • Performance on Hard Tasks: While discussion provided minimal gains on simple SNLI tasks, it offered a massive boost on the deceptive ANLI datasets, suggesting collaboration is most valuable for complex problems.
  • Improved Objection Ability: Prompting with discussion data significantly enhanced the system's ability to resist "gaslighting." The model's capacity to object to incorrect human opinions rose from approximately 55% to 80%.
  • Human-Like Reasoning: BERTScore analysis indicated that system-generated arguments were statistically aligned with supportive human logic, demonstrating that the models were learning valid reasoning patterns.

Limitations and Future Work

Despite these gains, the approach faces challenges. Discussion-based inference increases computational cost and latency due to longer input sequences and multiple turns. Additionally, models remain somewhat vulnerable to persistent human "gaslighting" and may amplify social biases present in their training data. Future research will focus on expanding this framework to other NLP tasks like summarization and improving the resilience of the system's logical backbone.

Summary updated

BibTeX
@inproceedings{kaneko2024solving,
  author = {Masahiro Kaneko and Graham Neubig and Naoaki Okazaki},
  title = {{Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach}},
  booktitle = {EACL (Findings)},
  year = {2024},
  url = {https://openreview.net/forum?id=QZxvnPGrBx},
}
Figure