Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up

Jiahao Yuan, Dehui Du, Hao Zhang, Zixiang Di, Usman Naseem

TL;DR

The paper introduces Reversal of Thought (RoT), a plug-and-play framework that enhances LLM logical reasoning by optimizing prompts during a pre-inference "warm-up" phase. By reverse-engineering task definitions from input-output examples and ranking them via pairwise self-evaluation, RoT achieves state-of-the-art reasoning accuracy while avoiding the heavy runtime computational overhead of multi-step frameworks like Tree-of-Thought.

What Problem Does This Paper Address?

While Large Language Models (LLMs) excel at many tasks, complex mathematical and logical reasoning remain major hurdles. Advanced multi-step prompting methods like Tree-of-Thought (ToT) and Graph-of-Thought (GoT) help mitigate these issues, but they come with a steep catch: they require multiple LLM queries per inference step, drastically inflating computational costs and latency during runtime. Conversely, cost-saving methods that rely on pre-defined, rigid templates sacrifice the flexibility needed to handle diverse, novel tasks.

The authors address this trade-off by asking: Can we achieve the high accuracy of advanced reasoning frameworks without the massive runtime inference cost?

Figure 1: Comparison between CoT (Yao et al., 2024; Besta et al., 2024; Yang et al., 2024a) and Reversal of Thought (RoT)

Methods

RoT shifts the heavy lifting from the live inference phase to a warm-up phase prior to batch processing. It consists of two primary components:

  1. Preference-Guided Reverse Reasoning (PGRR): Instead of asking the model to solve a problem forward, RoT feeds the model input-output demonstrations and asks it to work backward to generate task definitions and logical pseudocode.
  2. Warm-up: The LLM generates multiple candidate prompts based on the examples.
  3. Pairwise Evaluation & Ranking: Capitalizing on the model's internal preferences (shaped during RLHF training), the LLM performs a pairwise self-evaluation to score and rank these candidate prompts, selecting the optimal "LLM-taste" prompt.
  4. Cognitive Preference Manager (CPM): An offline embedding model evaluates the similarity between the original task definition and the newly generated reverse-reasoned prompt to identify the model's "knowledge boundaries."
  5. If the task is known (high similarity), it aggregates the solution logic.
  6. If the task is unknown (low similarity), it extracts a stylistic template from the context to safely guide the model without triggering hallucinations.
The architecture of the Reversal of Thought (RoT) framework

Datasets

The framework was evaluated across eight distinct tasks spanning five established logical and mathematical benchmarks:

  • Game of 24: Creating mathematical expressions from four numbers to equal 24.
  • BIG-Bench: Specifically focusing on Geometric Shapes, Multi-Step Arithmetic Two, Word Sorting, and Checkmate-in-One.
  • Python Puzzles: A dataset of challenging programming puzzles.
  • Multilingual Grade School Math (MGSM): A multilingual translation of grade-school math problems.
  • Shakespearean Sonnet Writing: Assessing literary creativity and constraint adherence.

Key Insights and Ideas

  • Leveraging "LLM-Taste" Prompts: LLMs have latent cognitive preferences shaped by Reinforcement Learning from Human Feedback (RLHF). RoT shows that models perform significantly better when executing instructions structured in their "own language" (e.g., structured logical pseudocode) rather than human-engineered prompts.
  • Amortizing the Cost of Thought: Rather than generating a complex reasoning tree for every single user query at runtime, RoT invests compute upfront during a warm-up phase to discover an optimal prompt. This prompt is then used efficiently across the entire batch of queries.
  • Reverse Engineering as a Boundary Detector: Forcing an LLM to deduce a problem's rules from its solution (reverse reasoning) is an effective way to map out what the model truly understands versus where it is guessing.

Key Results

RoT was tested using GPT-3.5-turbo and GPT-4 as backbone models, consistently outperforming classic Chain-of-Thought (CoT), Meta-Prompting, ToT, GoT, and Buffer-of-Thought (BoT) in both 1-shot and 2-shot settings.

  • Accuracy Gains: In the challenging Game of 24 benchmark using GPT-4, RoT achieved 97.0% accuracy in 1-shot and 98.0% in 2-shot settings, substantially outperforming the previous best baseline (BoT), which scored 82.8% and 83.7% respectively.
  • Efficiency Gains: As shown in the inference time breakdown, RoT dramatically reduces the time cost per sample compared to ToT and GoT, offering a highly competitive accuracy-to-cost ratio.
Figure 3: Inference time comparison, measured as the average duration from inference start to evaluation end, including all steps.

The complete performance breakdown across all datasets highlights RoT's consistent edge:

Detailed accuracy comparison table against baselines

Limitations and Future Work

The authors note that RoT's performance relies heavily on having high-quality, multi-shot demonstrations. In highly complex, multi-source tasks under strict 1-shot constraints, the reverse reasoning mechanism can occasionally struggle to accurately deduce the underlying logic.

To mitigate this, future work will explore integrating In-Context Learning (ICL) and automated demonstration selection strategies. Additionally, the authors suggest that applying RoT within a teacher-student model distillation framework could allow smaller, edge-deployed models to inherit these advanced reasoning capabilities.

Summary updated

Abstract

Large language models (LLMs) have shown remarkable performance in reasoning tasks but face limitations in mathematical and complex logical reasoning. Existing methods to improve LLMs' logical capabilities either involve traceable or verifiable logical sequences that generate more reliable responses by constructing logical structures yet increase computational costs, or introduces rigid logic template rules, reducing flexibility. In this paper, we propose Reversal of Thought (RoT), a plug-and-play and cost-effective reasoning framework designed to enhance the logical reasoning abilities of LLMs during the warm-up phase prior to batch inference. RoT utilizes a Preference-Guided Reverse Reasoning warm-up strategy, which integrates logical symbols for pseudocode planning through meta-cognitive mechanisms and pairwise preference self-evaluation to generate task-specific prompts solely through demonstrations, aligning with LLMs' cognitive preferences shaped by RLHF. Through reverse reasoning, we utilize a Cognitive Preference Manager to assess knowledge boundaries and further expand LLMs' reasoning capabilities by aggregating solution logic for known tasks and stylistic templates for unknown tasks. Experiments across various tasks demonstrate that RoT surpasses existing baselines in both reasoning accuracy and efficiency.
BibTeX
@inproceedings{yuan2025reversal,
  author = {Jiahao Yuan and Dehui Du and Hao Zhang and Zixiang Di and Usman Naseem},
  title = {{Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure