Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code

Keqin Bao, Nuo Chen, Xiaoyuan Li, Binyuan Hui, Bowen Yu, Fuli Feng, Xiangnan He, Dayiheng LiuarXiv 2025 · DOI · arXiv

Abstract

While code is a powerful medium for logical structure, forcing Large Language Models (LLMs) to strictly simulate code execution often results in suboptimal reasoning paths and overfitting to rigid syntax. We present TeaR (Teaching LLMs to Reason), a novel reinforcement learning framework that utilizes algorithmic problems to foster general-purpose reasoning. Unlike traditional methods that rely on Supervised Fine-Tuning (SFT) of code traces, TeaR employs Group Relative Policy Optimization (GRPO) to reward models for correct test-case outputs using their own natural language reasoning. Our approach yields substantial improvements across 17 benchmarks, including a 35.9% average gain on the Qwen2.5-7B base model.

Introduction: The Reasoning Gap in Code Execution

A common strategy for enhancing LLM reasoning involves training models to simulate code execution step-by-step. However, this paradigm suffers from a fundamental misalignment: code is optimized for computational efficiency and large-scale systems, whereas human-like reasoning should be flexible and token-efficient for small-scale problems.

Figure 3: A comparison between a model's ability to generate correct code versus its failure to predict the output of a simple test case for the same problem.

As illustrated in Figure 3, a model may successfully generate complex code to solve an algorithmic task but fail to manually execute that same logic on a simple input. This "reasoning gap" suggests that models often treat code as a memorized pattern rather than a logical tool they can internalize.

The TeaR Framework

To bridge this gap, TeaR shifts the objective from imitating code to solving the underlying logical task. The framework consists of two primary stages:

  1. Curated Algorithmic Data: We derived 200,000 input-output pairs from 2,400 distinct LeetCode problems. We applied rigorous filters to exclude guessable answers and overly complex cases that would exceed context limits, ensuring a high-quality signal for logical deduction.
  2. Reinforcement Learning via GRPO: Instead of providing code traces, the model is prompted with a problem description and a test case. It must generate its own reasoning path to reach the correct output. Rewards are based solely on the exact match of the final answer against ground-truth outputs. This encourages the model to explore and refine its latent reasoning capabilities without being tethered to specific coding syntax.

Key Insights and Results

The core insight of this work is that code logic does not always equal human logic. Forcing a model to trace an $O(n \log n)$ algorithm for a tiny array is unnatural and inefficient.

As shown in Figure 1, even state-of-the-art models like Qwen2.5-Coder exhibit a significant drop in accuracy when asked to predict test case outcomes directly compared to generating the code itself. TeaR addresses this by decoupling the logical requirement from the syntactic burden of code.

Performance Gains

The impact of TeaR is most evident in its performance across diverse benchmarks:

  • Base Model Elevation: Applying TeaR to Qwen2.5-7B resulted in a 35.9% average improvement across 17 benchmarks. Notably, MATH scores jumped from 49.8% to 75.2%.
  • Data Efficiency: TeaR achieved superior results in math compared to baselines like CodeI/O while using nearly 18 times less data (200K vs 3.5M samples).
  • Pushing the Ceiling: Even for models already optimized for long-chain-of-thought reasoning, such as DeepSeek-R1-Distilled-32B, TeaR provided an additional 5.9% performance boost.

Conclusion

TeaR demonstrates that algorithmic problems are a potent source of reasoning data, provided they are used to stimulate independent logical exploration rather than rote imitation. By rewarding verifiable outcomes through reinforcement learning, we can unlock latent reasoning capabilities that generalize far beyond the domain of computer science.

Summary updated

Abstract

Enhancing reasoning capabilities remains a central focus in the LLM reasearch community. A promising direction involves requiring models to simulate code execution step-by-step to derive outputs for given inputs. However, as code is often designed for large-scale systems, direct application leads to over-reliance on complex data structures and algorithms, even for simple cases, resulting in overfitting to algorithmic patterns rather than core reasoning structures. To address this, we propose TeaR, which aims at teaching LLMs to reason better. TeaR leverages careful data curation and reinforcement learning to guide models in discovering optimal reasoning paths through code-related tasks, thereby improving general reasoning abilities. We conduct extensive experiments using two base models and three long-CoT distillation models, with model sizes ranging from 1.5 billion to 32 billion parameters, and across 17 benchmarks spanning Math, Knowledge, Code, and Logical Reasoning. The results consistently show significant performance improvements. Notably, TeaR achieves a 35.9% improvement on Qwen2.5-7B and 5.9% on R1-Distilled-7B.
BibTeX
@misc{bao2025teaching,
  author = {Keqin Bao and Nuo Chen and Xiaoyuan Li and Binyuan Hui and Bowen Yu and Fuli Feng and Xiangnan He and Dayiheng Liu},
  title = {{Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code}},
  year = {2025},
  eprint = {2507.07498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.07498},
}
Figure