Abstract
Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), reflecting their ability to derive valid conclusions from given premises. While test-time scaling with reward models has enhanced performance in math and coding, its application to deductive logical reasoning remains under-explored. We present a set of Outcome Reward Models (ORMs) for deductive reasoning trained on standard Chain-of-Thought (CoT) data and a novel "echo generation" technique. This technique leverages LLMs' tendency to reflect incorrect assumptions in prompts to extract additional training data covering previously unexplored error types. Our results show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.
Introduction
While LLMs have shown impressive gains through test-time scaling—generating multiple solutions and using a reward model to select the best one—deductive logical reasoning remains a stubborn challenge. The primary bottleneck is the quality of data used to train verifiers. Standard datasets often lack "hard negatives": reasoning chains that appear logically sound on the surface but contain subtle fallacies. Without diverse examples of failure modes, reward models struggle to accurately rank candidate solutions, often falling prey to the same hallucinations as the reasoners they are meant to judge.
Methodology: Echo Chain-of-Thought (EcCoT)
To address the data scarcity for logical errors, we propose a two-pronged strategy for training robust ORMs:
- Multi-Sample CoT: Instead of training on a single reasoning path, we generate multiple samples per question, providing the reward model with a broader view of potential correct and incorrect trajectories.
- Echo Chain-of-Thought (EcCoT): This is our primary innovation. We exploit LLM sycophancy—the tendency to agree with user prompts—by deliberately prompting a model with an incorrect answer and asking it to "reason step-by-step" to justify that conclusion.
- Filtering via LLM Judge: To ensure high-quality training data, an LLM judge filters these "echoes." We retain only those where the reasoning appears logically consistent despite leading to the wrong conclusion, creating a dataset of deceptive reasoning paths.
At inference time, we employ Best-of-N sampling: the reasoner generates $N$ candidates, and the trained ORM assigns a score to each, selecting the highest-scored candidate as the final answer.
Experimental Results
We evaluated our approach across three major benchmarks: FOLIO (natural language first-order logic), ProverQA (multi-level logical questions), and JustLogic (synthetic logic with varying complexity).
Our experiments revealed that ORMs trained with EcCoT consistently outperformed both majority voting and standard CoT-trained ORMs. Notably, the quality of the reward model proved as vital as the number of samples. While standard models often plateaued, EcCoT-trained models continued to improve as the number of samples ($N$) increased.
Key Insights
- Sycophancy as a Feature: An LLM's tendency to agree with a wrong prompt is traditionally a weakness; here, it is a tool to generate effective "hard negatives."
- Diversity and Scaling: EcCoT produces significantly more diverse reasoning paths than standard CoT. This diversity prevents the reward model from over-fitting to narrow error patterns, allowing for better performance during test-time scaling.
- Teacher Models: Using a stronger model (e.g., GPT-4o) to generate training data for the reward model yielded superior results compared to using the reasoner's own generations, highlighting the value of high-quality "teacher" data in reward modeling.
Conclusion
This work demonstrates that training ORMs on a mixture of standard reasoning and "echoed" errors significantly boosts logical reasoning performance. By tricking models into justifying incorrect answers, we create a diverse dataset that teaches reward models to better distinguish valid logic from hallucinations. Future work may explore applying these techniques to Process Reward Models (PRMs) to grade individual reasoning steps, potentially closing the "reasoning gap" between generation and verification.
Summary updated