Logical Reasoning with Outcome Reward Models for Test-Time Scaling

Ramya Keerthy Thatikonda, Wray L. Buntine, Ehsan Shareghi

Abstract

Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), reflecting their ability to derive valid conclusions from given premises. While test-time scaling with reward models has enhanced performance in math and coding, its application to deductive logical reasoning remains under-explored. We present a set of Outcome Reward Models (ORMs) for deductive reasoning trained on standard Chain-of-Thought (CoT) data and a novel "echo generation" technique. This technique leverages LLMs' tendency to reflect incorrect assumptions in prompts to extract additional training data covering previously unexplored error types. Our results show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.

Introduction

While LLMs have shown impressive gains through test-time scaling—generating multiple solutions and using a reward model to select the best one—deductive logical reasoning remains a stubborn challenge. The primary bottleneck is the quality of data used to train verifiers. Standard datasets often lack "hard negatives": reasoning chains that appear logically sound on the surface but contain subtle fallacies. Without diverse examples of failure modes, reward models struggle to accurately rank candidate solutions, often falling prey to the same hallucinations as the reasoners they are meant to judge.

Methodology: Echo Chain-of-Thought (EcCoT)

To address the data scarcity for logical errors, we propose a two-pronged strategy for training robust ORMs:

  1. Multi-Sample CoT: Instead of training on a single reasoning path, we generate multiple samples per question, providing the reward model with a broader view of potential correct and incorrect trajectories.
  2. Echo Chain-of-Thought (EcCoT): This is our primary innovation. We exploit LLM sycophancy—the tendency to agree with user prompts—by deliberately prompting a model with an incorrect answer and asking it to "reason step-by-step" to justify that conclusion.
  3. Filtering via LLM Judge: To ensure high-quality training data, an LLM judge filters these "echoes." We retain only those where the reasoning appears logically consistent despite leading to the wrong conclusion, creating a dataset of deceptive reasoning paths.
The EcCoT pipeline (B) generates hard negative reasoning paths by forcing the model to justify incorrect answers, complementing standard CoT data (A).

At inference time, we employ Best-of-N sampling: the reasoner generates $N$ candidates, and the trained ORM assigns a score to each, selecting the highest-scored candidate as the final answer.

Experimental Results

We evaluated our approach across three major benchmarks: FOLIO (natural language first-order logic), ProverQA (multi-level logical questions), and JustLogic (synthetic logic with varying complexity).

Our experiments revealed that ORMs trained with EcCoT consistently outperformed both majority voting and standard CoT-trained ORMs. Notably, the quality of the reward model proved as vital as the number of samples. While standard models often plateaued, EcCoT-trained models continued to improve as the number of samples ($N$) increased.

Accuracy across ProverQA, JustLogic, and Folio datasets. EcCoT-augmented ORMs (green) consistently outperform standard CoT ORMs (orange) and Majority Voting (blue) as the number of samples scales.

Key Insights

  • Sycophancy as a Feature: An LLM's tendency to agree with a wrong prompt is traditionally a weakness; here, it is a tool to generate effective "hard negatives."
  • Diversity and Scaling: EcCoT produces significantly more diverse reasoning paths than standard CoT. This diversity prevents the reward model from over-fitting to narrow error patterns, allowing for better performance during test-time scaling.
  • Teacher Models: Using a stronger model (e.g., GPT-4o) to generate training data for the reward model yielded superior results compared to using the reasoner's own generations, highlighting the value of high-quality "teacher" data in reward modeling.

Conclusion

This work demonstrates that training ORMs on a mixture of standard reasoning and "echoed" errors significantly boosts logical reasoning performance. By tricking models into justifying incorrect answers, we create a diverse dataset that teaches reward models to better distinguish valid logic from hallucinations. Future work may explore applying these techniques to Process Reward Models (PRMs) to grade individual reasoning steps, potentially closing the "reasoning gap" between generation and verification.

Summary updated

Abstract

Logical reasoning is a critical benchmark for evaluating the capabilities of large language models (LLMs), as it reflects their ability to derive valid conclusions from given premises. While the combination of test-time scaling with dedicated outcome or process reward models has opened up new avenues to enhance LLMs performance in complex reasoning tasks, this space is under-explored in deductive logical reasoning. We present a set of Outcome Reward Models (ORMs) for deductive reasoning. To train the ORMs we mainly generate data using Chain-of-Thought (CoT) with single and multiple samples. Additionally, we propose a novel tactic to further expand the type of errors covered in the training dataset of the ORM. In particular, we propose an echo generation technique that leverages LLMs' tendency to reflect incorrect assumptions made in prompts to extract additional training data, covering previously unexplored error types. While a standard CoT chain may contain errors likely to be made by the reasoner, the echo strategy deliberately steers the model toward incorrect reasoning. We show that ORMs trained on CoT and echo-augmented data demonstrate improved performance on the FOLIO, JustLogic, and ProverQA datasets across four different LLMs.
BibTeX
@inproceedings{thatikonda2025logical,
  author = {Ramya Keerthy Thatikonda and Wray L. Buntine and Ehsan Shareghi},
  title = {{Logical Reasoning with Outcome Reward Models for Test-Time Scaling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=RGxAwzatHv},
}
Figure