RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

Qianyue Hao, Sibo Li, Jian Yuan, Yong Li

TL;DR

Current inference-time reasoning techniques for LLMs (like Chain-of-Thought) rely on rigid, hand-crafted structures that fail to adapt to specific problems. RL-of-Thoughts (RLoT) solves this by training an ultra-lightweight (<3K parameters) reinforcement learning "navigator" that dynamically builds task-specific reasoning pathways at inference time. This tiny model boosts sub-10B models to perform on par with 100B+ giants while remaining highly transferable across different tasks and models.

What Problem Does This Paper Address?

Large Language Models (LLMs) generate text token-by-token in an autoregressive manner, making complex, multi-step reasoning inherently difficult. To mitigate this, popular inference-time strategies like Chain-of-Thought (CoT) or Tree-of-Thoughts (ToT) guide models through structured thinking.

However, these methods are static and task-agnostic—they apply the same rigid logical structure uniformly, whether solving a simple word problem or a complex Olympiad math puzzle. While advanced tree-search methods (like Monte Carlo Tree Search) offer adaptability, they require extensive trial-and-error rollouts, making them computationally expensive and slow. The authors address a critical question: How can we dynamically and efficiently tailor the logical reasoning structure to a specific problem at inference time without modifying the underlying LLM's parameters?

Methods

The authors introduce RL-of-Thoughts (RLoT), framing long-sequence reasoning as a Markov Decision Process (MDP). Instead of fine-tuning the massive LLM itself, they train a tiny, 3-layer Multilayer Perceptron (MLP) with a Dueling Network architecture (containing just 2,566 parameters) called the navigator model.

  • State Space: The current problem-solving status is summarized into a low-dimensional vector using an LLM self-evaluation prompt across three major dimensions: Correctness, Complexity, and Completeness.
  • Action Space: The navigator chooses from five human-cognition-inspired "basic logic blocks": Reason one step, Decompose, Debate, Refine, and Terminate.
  • Reward: A frozen Process Reward Model (PRM) evaluates the quality of intermediate steps after an action is taken, providing a dense reward signal.

During inference, the trained RL navigator acts as a guide, dynamically selecting and cascading these logic blocks based on the problem's evolving state to build a customized reasoning pathway.

Figure 1: Framework of RL-of-Thoughts (RLoT). We train an RL agent as the navigator, which dynamically selects and combines basic logic blocks along the reasoning process, constructing task-specific logical structures for each task and thereby enhancing the LLMs’ ability to handle complex reasoning tasks.

Datasets

The framework was evaluated across a diverse suite of challenging benchmarks:

  • Mathematics: AIME24 (Olympiad-level), AMC23, GSM8K (elementary), and MATH.
  • STEM: MMLU-STEM and GPQA (graduate-level, "Google-proof" questions).
  • Commonsense Reasoning: StrategyQA (multi-hop questions requiring implicit strategies).

Key Insights and Ideas

  • Extreme Parameter Efficiency: The navigator model requires fewer than 3,000 parameters. Remarkably, this tiny RL agent can bridge the performance gap, allowing sub-10B parameter models to rival or exceed 100B+ parameter giants.
  • Remarkable Transferability: A navigator trained on a specific LLM (e.g., Qwen) and task (e.g., MATH) generalizes incredibly well to completely unseen LLMs (like Llama or GPT) and different domains (like STEM or commonsense) without any additional fine-tuning.
  • PRM Generalization: Process Reward Models trained purely on mathematical data provide highly effective, directionally accurate reward signals for non-math domains like GPQA and StrategyQA, explaining why the navigator transfers so well.
  • Direct Trajectory Generation vs. Search: Unlike tree-search methods that rely on expensive trial-and-error rollouts, RLoT directly generates the optimal logical path, drastically reducing token consumption and latency.

Key Results

  • RLoT consistently outperformed standard inference-time baselines (Direct QA, CoT, CoT-SC, ToT) across multiple base models including Qwen2.5, Llama3.1, GPT-4o-mini, and DeepSeek-R1-Distill.
  • On the highly challenging GPQA benchmark, RLoT achieved up to a 13.4% performance boost when paired with Llama3.1-8B-Instruct.
  • In terms of efficiency, RLoT's token consumption is comparable to Chain-of-Thought with Self-Consistency (CoT-SC) and consumes only about 78% of the tokens required by Tree-of-Thoughts (ToT).

Overall evaluation of RLoT's capability across different tasks Evaluation of RLoT's transferability across different LLMs

Limitations and Future Work

  • Self-Evaluation Noise: The framework relies on the base LLM to evaluate its own intermediate steps. If the LLM hallucinates that an incorrect calculation is "correct," it passes faulty state information to the navigator, leading to suboptimal actions (e.g., choosing "Debate" instead of "Refine").
  • Action Selection Errors: Occasionally, the navigator selects a suboptimal action—such as triggering an unnecessary "Refine" step on a perfectly correct equation—which can lead to "over-correction" and final answer failure.
  • Expanding the Action Space: The authors suggest that future work could explore designing and integrating more diverse reasoning blocks (such as rephrasing or alternative cognitive strategies) to further expand RLoT's capabilities.

Summary updated

Abstract

Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities. To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly cost-effective by guiding reasoning through sophisticated logical structures without modifying LLMs' parameters. However, these manually predefined, task-agnostic frameworks are applied uniformly across diverse tasks, lacking adaptability. To improve this, we propose RL-of-Thoughts (RLoT), where we train a lightweight navigator model with reinforcement learning (RL) to adaptively enhance LLM reasoning at inference time. Specifically, we design five basic logic blocks from the perspective of human cognition. During the reasoning process, the trained RL navigator dynamically selects the suitable logic blocks and combines them into task-specific logical structures according to problem characteristics. Experiments across multiple reasoning benchmarks (AIME, MATH, GPQA, etc.) with multiple LLMs (GPT, Llama, Qwen, and DeepSeek) illustrate that RLoT outperforms established inference-time techniques by up to 13.4%. Remarkably, with less than 3K parameters, our RL navigator is able to make sub-10B LLMs comparable to 100B-scale counterparts. Moreover, the RL navigator demonstrates strong transferability: a model trained on one specific LLM-task pair can effectively generalize to unseen LLMs and tasks. Our code is open-source at https://github.com/tsinghua-fib-lab/RL-LLM-Reasoning for reproducibility.
BibTeX
@inproceedings{hao2026rl,
  author = {Qianyue Hao and Sibo Li and Jian Yuan and Yong Li},
  title = {{RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=Dw034qKrP5},
}
Figure