Debating with More Persuasive LLMs Leads to More Truthful Answers

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, Ethan Perez

Abstract

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

Introduction

As Large Language Models (LLMs) approach and eventually exceed human expertise in specialized domains, a critical "supervision gap" emerges. If a model possesses knowledge beyond its human evaluator, traditional alignment techniques based on human labeling become unreliable. This challenge, known as scalable oversight, necessitates new protocols where non-experts can effectively supervise experts. This paper explores AI Debate as a solution, testing the hypothesis that "the truth is a competitive advantage"—that it is inherently easier to argue for a correct position than a false one under scrutiny.

Experimental Methodology

The study utilizes an information-asymmetric environment based on the QuALITY dataset, specifically its "HARD" subset of long-document reading comprehension questions.

Setup and Protocols

The researchers designated strong models (e.g., GPT-4-Turbo) as "Experts" with access to the full source text (Project Gutenberg science fiction stories). "Judges" (humans or weaker LLMs) were denied access to the text and had to decide the correct answer based on expert input. Three primary protocols were evaluated:

  1. Consultancy: A single expert argues for one answer (either correct or incorrect).
  2. Debate: Two experts argue for opposing answers simultaneously.
  3. Interactive Debate: The judge can actively participate by asking clarifying questions.

To ensure factual grounding, experts used a "Quote Tool" to provide verified snippets from the text. Debaters were optimized for persuasiveness using Best-of-N sampling and Critique-and-Refinement.

Overview of the AI Debate protocol and experimental setup, illustrating the information asymmetry between LLM experts and non-expert judges.

Key Results

The empirical results strongly support the efficacy of the debate protocol in bridging the supervision gap.

Performance Gains

Debate significantly improved the accuracy of non-expert judges. Human judges reached 88% accuracy (up from a 60% naive baseline), while LLM judges reached 76% accuracy (up from 48%). This represents a recovery of approximately 60% of the performance gap between a non-expert and an expert judge with full text access.

Judge accuracy across different protocols and judge models. Debate consistently outperforms consultancy and naive baselines across all judge types.

Persuasiveness and Truth

A crucial finding is the relationship between debater skill and judge accuracy. As experts were optimized to be more persuasive (increasing their Elo rating), judge accuracy in the debate protocol increased. Conversely, in the consultancy protocol, increased persuasiveness led to lower judge accuracy, as the single expert could more easily mislead the judge. This suggests that competitive pressure in a debate forces models to rely on the truth to remain persuasive.

Discussion and Future Work

While debate shows great promise, several challenges remain. The current study focused on an information gap; future research must determine if these findings translate to reasoning gaps (e.g., mathematics or code). Additionally, the reliance on a quote system for evidence verification is a key component that needs adaptation for non-textual domains. Despite these limitations, the results suggest that LLM-generated debates can significantly scale human oversight of complex AI systems, providing a path toward aligning superhuman intelligence.

Summary updated

Abstract

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.
BibTeX
@inproceedings{khan2024debating,
  author = {Akbir Khan and John Hughes and Dan Valentine and Laura Ruis and Kshitij Sachan and Ansh Radhakrishnan and Edward Grefenstette and Samuel R. Bowman and Tim Rocktäschel and Ethan Perez},
  title = {{Debating with More Persuasive LLMs Leads to More Truthful Answers}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=iLCZtl7FTa},
}
Figure