TL;DR
LoT (Logical Thoughts) is a new prompting framework that significantly boosts the reasoning accuracy of Large Language Models (LLMs) by integrating symbolic logic into the reasoning process. By using a "think-verify-revise" loop based on the principle of Reductio ad Absurdum (proof by contradiction), LoT allows models to systematically catch and correct their own hallucinations in a zero-shot setting.
What Problem Does This Paper Address?
While LLMs like GPT-4 are impressively knowledgeable, they frequently stumble on tasks requiring multi-step reasoning. A common failure mode is the "hallucination chain": the model makes a small logical error in an early step, and because standard Chain-of-Thought (CoT) prompting is an open-loop process, that error propagates, leading to a confidently stated but entirely wrong conclusion.
The authors identify that LLMs possess logical "tricks" but fail to use them in an organized way. The gap they tackle is the lack of a formal, self-correcting thinking paradigm that constrains model outputs to logical principles.
Methods
The core of the paper is the LoT (Logical Thoughts) framework, which transforms reasoning from a straight line into a self-improving loop.
- Reductio ad Absurdum: To verify a reasoning step ($T_i$), the model doesn't just ask "is this right?" Instead, it follows a logical proof by contradiction. It assumes the negation of the step ($\neg T_i$) and checks if that leads to a contradiction with the original premises.
- Think-Verify-Revise Loop:
- Think: The model generates a reasoning step.
- Verify (Adpt-LoT): The model generates two "post-hoc" explanations: one arguing why the step is true ($E_i$) and one arguing why it is false ($E_{\neg i}$). It then acts as a discriminator to choose the more plausible explanation.
- Revise: If the model decides the step was likely false, it discards all subsequent steps, uses the "why it was wrong" explanation as feedback, and regenerates a corrected reasoning chain.
This approach turns verification into a preference discrimination task, which LLMs are generally better at than open-ended self-correction.
Datasets
The researchers evaluated LoT across a diverse battery of benchmarks to test different types of reasoning:
- Arithmetic: GSM8K (grade school math) and AQuA (algebra word problems).
- Commonsense: DateUnderstanding and OddOneOut (identifying the unusual object).
- Causal Inference: CauseEffect and ShuffledObjects.
- Symbolic Reasoning: LastLetter (concatenating the last letters of words).
- Social Intelligence: SocialQA (emotional and social intelligence in daily activities).
Key Insights and Ideas
- The Power of Opposing Viewpoints: Forcing the model to argue against its own generated step (generating $E_{\neg i}$) is a powerful "compulsory" error-detection mechanism. It breaks the model's inherent bias toward its own previous outputs.
- Scaling Logical Competence: The effectiveness of LoT scales with the model's size. While smaller models (like Vicuna-7b) sometimes struggle to follow the complex verification instructions, larger models like GPT-4 show robust self-correction abilities.
- Concise Reasoning: Interestingly, the authors found that LoT-refined chains are often slightly shorter than original CoT chains. The verification process seems to encourage the model to discard fluff and focus on information-dense, logically sound steps.
- Closing the Loop: Standard CoT is "fragile"—one broken link ruins the chain. LoT makes the chain "resilient" by verifying each link before moving forward.
Key Results
LoT consistently outperformed standard zero-shot CoT across almost all models and datasets.
- GPT-4 Performance: GPT-4 saw steady gains, particularly in math (+1.42% on GSM8K) and date understanding (+2.07%). It achieved perfect 100% scores on causal inference tasks.
- Vicuna Improvements: The Vicuna-13b model saw a massive 16.28% accuracy jump on the OddOneOut task when enhanced with LoT.
- Ablation Success: The "Adaptive" version of LoT (Adpt-LoT) outperformed simpler versions that didn't use the dual-explanation strategy, proving that the "Reductio ad Absurdum" framing is critical for catching subtle errors.
Limitations and Future Work
- Computational Cost: Because LoT requires multiple calls to the LLM for every reasoning step (to generate explanations and verify them), it is significantly slower and more expensive in terms of token usage than standard CoT.
- Small Model Fragility: As seen in the "worsening rates" for Vicuna-7b, very small models can sometimes be "fooled" by their own false explanations, leading them to correct a right answer into a wrong one.
- Future Directions: The authors suggest exploring fine-tuning LLMs specifically to perform this logical verification spontaneously. They also see potential in using LoT-verified reasoning chains as high-quality data for Reinforcement Learning from AI Feedback (RLAIF).
Summary updated