Stimulate the Critical Thinking of LLMs via Debiasing Discussion

Ruiyu Xiao, Lei Wu, Yuanxing Liu, Weinan Zhang, Ting LiuEMNLP 2025

TL;DR

Large language models (LLMs) often struggle with critical thinking in discussions, tending to agree with users or get stuck in biased viewpoints. This paper introduces MDTA, a novel training framework that debiases LLMs by exposing them to multi-stance discussions and aligning them with ground truth, significantly improving their critical reasoning and performance on downstream tasks.

What Problem Does This Paper Address?

LLMs, despite their impressive capabilities, exhibit a concerning lack of critical thinking when engaged in discussions. This manifests in two key biases:

  1. Stance Homogeneity Bias: LLM training data often lacks diverse perspectives, leading models to struggle when encountering conflicting viewpoints. They tend to get disrupted and fail to reach coherent conclusions.
  2. Human Preference Bias: The prevalent Reinforcement Learning from Human Feedback (RLHF) paradigm can inadvertently amplify human biases. Humans may reject viewpoints that challenge their own, causing LLMs to overly prioritize user opinions, even incorrect ones, rather than objective truth.

These biases lead to LLMs that are not truly critical thinkers, hindering their ability to engage in productive, truth-seeking discussions and potentially leading to flawed decision-making in real-world applications.

Methods

The paper proposes a novel two-stage training framework called MDTA (Multi-stance Discussion Sampling and Truth Alignment Training) to address these biases:

  • Multi-stance Discussion Sampling (MD): This stage tackles stance homogeneity bias. The authors introduce an "egalitarian and comprehensive multiagent discussion framework" to automatically generate a large-scale, unbiased discussion dataset named Eq-Discussion. This framework simulates brainstorming, where agents take on roles of proponents, challengers, and summarizers, generating diverse stances and counterarguments.
  • Truth Alignment Training (TA): This stage addresses human preference bias. Instead of relying on human preference data for RLHF, MDTA uses a modified approach. It employs the KTO (Kahneman-Tversky Optimization) algorithm, which aligns the model with "ground truth" rather than human preferences. This involves a rule-based acceptability function that labels data samples based on whether the model produces the correct answer, encouraging critical examination of user perspectives.

Datasets

  • Eq-Discussion: The authors introduce this as the first and largest multi-model fair discussion dataset, containing over 200,000 dialogues and 100 million tokens. It's generated using their multi-stance discussion framework.
  • Evaluation Datasets: The experiments utilize four standard datasets:
  • MMLU: For knowledge QA.
  • OpenBookQA: For reading comprehension.
  • CommonsenseQA: For logical reasoning.
  • Social IQa: For social interaction abilities.

Key Insights and Ideas

  • LLMs are not inherently critical: The paper provides strong evidence that current LLMs suffer from significant biases that impede their critical thinking abilities in discussion settings.
  • Bias stems from training data and methods: Both the lack of diverse viewpoints in training data (stance homogeneity) and the reliance on human preferences in RLHF (human preference bias) are identified as root causes.
  • Multi-stance discussion is crucial: Simply exposing LLMs to more data isn't enough; it needs to be data that actively encourages debate and challenges viewpoints.
  • Ground truth alignment is more robust than human preference: Shifting RLHF from human preference to ground truth alignment is key to overcoming sycophancy and promoting objective reasoning.
  • MDTA generalizes well: The improvements seen with MDTA are not limited to discussion scenarios; they also translate to better performance on general downstream tasks and out-of-domain datasets.

Key Results

  • Reduced Human Preference Bias: MDTA improved the Correct Agreement Rate (CAR) by an average of 58.5 points across open-source models and benchmarks. Models trained with MDTA showed higher agreement with correct user responses (Agree(Dcorr) improved by 16.0 points) and lower agreement with incorrect ones (Agree(Dincorr) decreased by 43.7 points).
  • Reduced Stance Homogeneity Bias: The Discussion Result Accuracy (DRA) metric showed consistent improvements, with MDTA-trained models achieving higher accuracy in self-discussion scenarios, improving the DAR metric by 10.25 points on average.
  • State-of-the-Art Performance: MDTA achieved SOTA results in reading comprehension, logical reasoning, and social QA through self-discussion.
  • Generalization: MDTA enhanced generalization capabilities, boosting performance in non-discussion scenarios and on out-of-domain data.

Limitations and Future Work

  • Theoretical Analysis of RLHF: The authors acknowledge a lack of deeper theoretical analysis regarding the reinforcement learning methods used in the RLHF phase. They plan to explore human preference correction in RLHF more deeply in future work.
  • Residual Bias: While MDTA significantly reduces bias, some residual issues remain. This is partly attributed to the dataset size and the difficulty of completely eliminating bias introduced during pre-training.
  • Challenging Error Types: The paper notes that certain error types, particularly those requiring the generation of entirely new answers during discussion (type (c)), are more challenging and less effectively handled by methods relying on self-voting. The MDTA approach shows promise here but still faces challenges.

Summary updated

BibTeX
@inproceedings{xiao2025stimulate,
  author = {Ruiyu Xiao and Lei Wu and Yuanxing Liu and Weinan Zhang and Ting Liu},
  title = {{Stimulate the Critical Thinking of LLMs via Debiasing Discussion}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure