Abstract
Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), reflecting their ability to derive valid conclusions from given premises. While test-time scaling with reward models has enhanced performance in math and coding, its application to deductive logic remains under-explored. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train these ORMs, we utilize Chain-of-Thought (CoT) data and introduce a novel "echo generation" technique (EcCoT). This tactic leverages LLMs’ tendency to reflect incorrect assumptions in prompts to generate "hard negatives"—plausible but flawed reasoning paths. We demonstrate that ORMs trained on CoT and echo-augmented data significantly improve performance on the FOLIO, JustLogic, and ProverQA datasets across multiple LLM families.
Introduction
Large Language Models (LLMs) have demonstrated remarkable capabilities, yet deductive logical reasoning remains a challenging frontier. While techniques like test-time scaling—generating multiple candidate answers and selecting the best via a reward model—have succeeded in other domains, logical reasoning suffers from a lack of high-quality, diverse training data for reward models. Specifically, it is difficult to obtain "hard negatives": reasoning chains that appear logically structured but lead to incorrect conclusions. Standard Chain-of-Thought (CoT) sampling often produces errors that are either too random or too obvious, failing to prepare reward models for the subtle fallacies encountered during inference.
Methodology: Echo Chain-of-Thought (EcCoT)
To address the data bottleneck, we propose a framework for training Outcome Reward Models (ORMs) specifically for deductive logic. Our approach combines multi-sample training with a novel data augmentation strategy called Echo Chain-of-Thought (EcCoT).
EcCoT exploits the known tendency of LLMs to "echo" or agree with user premises, even when incorrect. By prompting a generator model with a deliberately wrong answer (e.g., "Given the answer is True, please reason step-by-step..." when the gold answer is False), we force the model to justify a logical error. These "echoed" trajectories are then filtered by an LLM judge to ensure they are plausible but flawed, providing the ORM with the challenging negative examples necessary for robustness. At test-time, these ORMs are used for Best-of-N sampling, selecting the most confident candidate from $N$ generated reasoning paths.
Experimental Evaluation
We evaluated our approach using three diverse logical reasoning benchmarks:
- FOLIO: Human-annotated deductive reasoning tasks.
- ProverQA: Logical questions across easy, medium, and hard difficulty levels.
- JustLogic: A synthetic dataset testing deep reasoning (up to 7 layers of logic).
Results and Analysis
Our results demonstrate that ORMs trained on augmented EcCoT data consistently outperform standard CoT-trained ORMs and majority-voting baselines across all datasets and model families, including Qwen, Llama, and GPT-4o.
Key findings include:
- Consistent Scaling: Performance generally improves as the number of test-time samples ($N$) increases, with the EcCoT-augmented ORM maintaining a lead over other methods.
- Diversity over Quantity: Adding a small number of "Echo" samples proved more effective than adding a much larger volume of standard CoT samples, suggesting that the quality and type of error are paramount for reward model training.
- Impact on Smaller Models: While large models show steady gains, smaller models benefit significantly from ORM verification. For instance, models like Gemma-1B saw accuracy improvements of nearly 30% when paired with an EcCoT-trained ORM.
Discussion and Limitations
By weaponizing model weaknesses—specifically sycophancy—we turn a known flaw into a tool for data engineering. However, several challenges remain. Our current approach uses Outcome Reward Models, which verify only the final answer and may occasionally reward "right for the wrong reasons." Future work could extend the Echo technique to Process Reward Models (PRMs) to verify individual logical steps. Additionally, the effectiveness of EcCoT depends on the generator's ability to produce coherent text; if the generator is too weak, the resulting echoes may lack the structural plausibility required for effective training.
Conclusion
This work demonstrates that deliberately inducing and then learning from logical errors is a powerful strategy for improving LLM reasoning. By training ORMs on "echoed" data, we enable more effective test-time scaling, pushing the boundaries of deductive logic in both large and small language models.
Summary updated