Abstract
While code is a powerful medium for logical structure, forcing Large Language Models (LLMs) to strictly simulate code execution often results in suboptimal reasoning paths and overfitting to rigid syntax. We present TeaR (Teaching LLMs to Reason), a novel reinforcement learning framework that utilizes algorithmic problems to foster general-purpose reasoning. Unlike traditional methods that rely on Supervised Fine-Tuning (SFT) of code traces, TeaR employs Group Relative Policy Optimization (GRPO) to reward models for correct test-case outputs using their own natural language reasoning. Our approach yields substantial improvements across 17 benchmarks, including a 35.9% average gain on the Qwen2.5-7B base model.
Introduction: The Reasoning Gap in Code Execution
A common strategy for enhancing LLM reasoning involves training models to simulate code execution step-by-step. However, this paradigm suffers from a fundamental misalignment: code is optimized for computational efficiency and large-scale systems, whereas human-like reasoning should be flexible and token-efficient for small-scale problems.
As illustrated in Figure 3, a model may successfully generate complex code to solve an algorithmic task but fail to manually execute that same logic on a simple input. This "reasoning gap" suggests that models often treat code as a memorized pattern rather than a logical tool they can internalize.
The TeaR Framework
To bridge this gap, TeaR shifts the objective from imitating code to solving the underlying logical task. The framework consists of two primary stages:
- Curated Algorithmic Data: We derived 200,000 input-output pairs from 2,400 distinct LeetCode problems. We applied rigorous filters to exclude guessable answers and overly complex cases that would exceed context limits, ensuring a high-quality signal for logical deduction.
- Reinforcement Learning via GRPO: Instead of providing code traces, the model is prompted with a problem description and a test case. It must generate its own reasoning path to reach the correct output. Rewards are based solely on the exact match of the final answer against ground-truth outputs. This encourages the model to explore and refine its latent reasoning capabilities without being tethered to specific coding syntax.
Key Insights and Results
The core insight of this work is that code logic does not always equal human logic. Forcing a model to trace an $O(n \log n)$ algorithm for a tiny array is unnatural and inefficient.
As shown in Figure 1, even state-of-the-art models like Qwen2.5-Coder exhibit a significant drop in accuracy when asked to predict test case outcomes directly compared to generating the code itself. TeaR addresses this by decoupling the logical requirement from the syntactic burden of code.
Performance Gains
The impact of TeaR is most evident in its performance across diverse benchmarks:
- Base Model Elevation: Applying TeaR to Qwen2.5-7B resulted in a 35.9% average improvement across 17 benchmarks. Notably, MATH scores jumped from 49.8% to 75.2%.
- Data Efficiency: TeaR achieved superior results in math compared to baselines like CodeI/O while using nearly 18 times less data (200K vs 3.5M samples).
- Pushing the Ceiling: Even for models already optimized for long-chain-of-thought reasoning, such as DeepSeek-R1-Distilled-32B, TeaR provided an additional 5.9% performance boost.
Conclusion
TeaR demonstrates that algorithmic problems are a potent source of reasoning data, provided they are used to stimulate independent logical exploration rather than rote imitation. By rewarding verifiable outcomes through reinforcement learning, we can unlock latent reasoning capabilities that generalize far beyond the domain of computer science.
Summary updated