Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic

Xufeng Zhao, Mengdi Li, Wenhao Lu, Cornelius Weber, Jae Hee Lee, Kun Chu, Stefan Wermter

TL;DR

LoT (Logical Thoughts) is a new prompting framework that significantly boosts the reasoning accuracy of Large Language Models (LLMs) by integrating symbolic logic into the reasoning process. By using a "think-verify-revise" loop based on the principle of Reductio ad Absurdum (proof by contradiction), LoT allows models to systematically catch and correct their own hallucinations in a zero-shot setting.

What Problem Does This Paper Address?

While LLMs like GPT-4 are impressively knowledgeable, they frequently stumble on tasks requiring multi-step reasoning. A common failure mode is the "hallucination chain": the model makes a small logical error in an early step, and because standard Chain-of-Thought (CoT) prompting is an open-loop process, that error propagates, leading to a confidently stated but entirely wrong conclusion.

The authors identify that LLMs possess logical "tricks" but fail to use them in an organized way. The gap they tackle is the lack of a formal, self-correcting thinking paradigm that constrains model outputs to logical principles.

Methods

The core of the paper is the LoT (Logical Thoughts) framework, which transforms reasoning from a straight line into a self-improving loop.

  1. Reductio ad Absurdum: To verify a reasoning step ($T_i$), the model doesn't just ask "is this right?" Instead, it follows a logical proof by contradiction. It assumes the negation of the step ($\neg T_i$) and checks if that leads to a contradiction with the original premises.
  2. Think-Verify-Revise Loop:
    • Think: The model generates a reasoning step.
  3. Verify (Adpt-LoT): The model generates two "post-hoc" explanations: one arguing why the step is true ($E_i$) and one arguing why it is false ($E_{\neg i}$). It then acts as a discriminator to choose the more plausible explanation.
  4. Revise: If the model decides the step was likely false, it discards all subsequent steps, uses the "why it was wrong" explanation as feedback, and regenerates a corrected reasoning chain.
Figure 2: A diagram demonstrating the think-verify- revision loop of LoT. The two zoom-in boxes display the processes when a thought passes (top-left) and fails (bottom) the verification respectively. A thought passing the verification is kept in the rea- soning trace, while a thought failing the verification is revised and a new chain of thought is generated based on the revision. The symbols in this figure are introduced in Sec. 3.2 and Sec. 3.3. See also Fig. 4 in Appendix D with extended details.

This approach turns verification into a preference discrimination task, which LLMs are generally better at than open-ended self-correction.

Datasets

The researchers evaluated LoT across a diverse battery of benchmarks to test different types of reasoning:

  • Arithmetic: GSM8K (grade school math) and AQuA (algebra word problems).
  • Commonsense: DateUnderstanding and OddOneOut (identifying the unusual object).
  • Causal Inference: CauseEffect and ShuffledObjects.
  • Symbolic Reasoning: LastLetter (concatenating the last letters of words).
  • Social Intelligence: SocialQA (emotional and social intelligence in daily activities).

Key Insights and Ideas

  • The Power of Opposing Viewpoints: Forcing the model to argue against its own generated step (generating $E_{\neg i}$) is a powerful "compulsory" error-detection mechanism. It breaks the model's inherent bias toward its own previous outputs.
  • Scaling Logical Competence: The effectiveness of LoT scales with the model's size. While smaller models (like Vicuna-7b) sometimes struggle to follow the complex verification instructions, larger models like GPT-4 show robust self-correction abilities.
  • Concise Reasoning: Interestingly, the authors found that LoT-refined chains are often slightly shorter than original CoT chains. The verification process seems to encourage the model to discard fluff and focus on information-dense, logically sound steps.
  • Closing the Loop: Standard CoT is "fragile"—one broken link ruins the chain. LoT makes the chain "resilient" by verifying each link before moving forward.

Key Results

LoT consistently outperformed standard zero-shot CoT across almost all models and datasets.

  • GPT-4 Performance: GPT-4 saw steady gains, particularly in math (+1.42% on GSM8K) and date understanding (+2.07%). It achieved perfect 100% scores on causal inference tasks.
  • Vicuna Improvements: The Vicuna-13b model saw a massive 16.28% accuracy jump on the OddOneOut task when enhanced with LoT.
  • Ablation Success: The "Adaptive" version of LoT (Adpt-LoT) outperformed simpler versions that didn't use the dual-explanation strategy, proving that the "Reductio ad Absurdum" framing is critical for catching subtle errors.
Accuracy results comparing CoT and LoT across different models and datasets.

Limitations and Future Work

  • Computational Cost: Because LoT requires multiple calls to the LLM for every reasoning step (to generate explanations and verify them), it is significantly slower and more expensive in terms of token usage than standard CoT.
  • Small Model Fragility: As seen in the "worsening rates" for Vicuna-7b, very small models can sometimes be "fooled" by their own false explanations, leading them to correct a right answer into a wrong one.
  • Future Directions: The authors suggest exploring fine-tuning LLMs specifically to perform this logical verification spontaneously. They also see potential in using LoT-verified reasoning chains as high-quality data for Reinforcement Learning from AI Feedback (RLAIF).

Summary updated

Abstract

Recent advancements in large language models have showcased their remarkable generalizability across various domains. However, their reasoning abilities still have significant room for improvement, especially when confronted with scenarios requiring multi-step reasoning. Although large language models possess extensive knowledge, their reasoning often fails to effectively utilize this knowledge to establish a coherent thinking paradigm. These models sometimes show hallucinations as their reasoning procedures are unconstrained by logical principles. Aiming at improving the zero-shot chain-of-thought reasoning ability of large language models, we propose LoT (Logical Thoughts), a self-improvement prompting framework that leverages principles rooted in symbolic logic, particularly Reductio ad Absurdum, to systematically verify and rectify the reasoning processes step by step. Experimental evaluations conducted on language tasks in diverse domains, including arithmetic, commonsense, symbolic, causal inference, and social problems, demonstrate the efficacy of enhanced reasoning by logic. The implementation code for LoT can be accessed at: https://github.com/xf-zhao/LoT.
BibTeX
@inproceedings{zhao2024enhancing,
  author = {Xufeng Zhao and Mengdi Li and Wenhao Lu and Cornelius Weber and Jae Hee Lee and Kun Chu and Stefan Wermter},
  title = {{Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=BBB9eR7RSV},
}
Figure