START: Self-taught Reasoner with Tools

Cheng‐Peng Li, Mingfeng Xue, Zhenru Zhang, Jiaxi Yang, Beichen Zhang, Bowen Yu, Binyuan Hui, Junyang Lin, Xiang Wang, Dayiheng Liu2025 · DOI

Abstract

Large Reasoning Models (LRMs) excel at complex logic but often stumble over basic arithmetic and algorithmic execution. START (Self-taught Reasoner with Tools) introduces a framework that activates a model’s latent ability to use a Python interpreter through "artificial hints" and a self-improving training loop. The result is a model that is not only more accurate on math and science benchmarks but also more efficient, replacing verbose text-based reasoning with concise, precise code.

Introduction

Current state-of-the-art reasoning models, such as DeepSeek-R1 or OpenAI’s o1, utilize long Chain-of-Thought (CoT) to solve difficult problems. However, these models are essentially "calculators made of words"—they frequently hallucinate numerical results or fail at symbolic manipulation. While "Tool-Integrated Reasoning" (TIR) isn't new, the challenge lies in teaching a model to use tools during its internal monologue without degrading its general reasoning strengths.

The authors identify a "loss of generalization" in instruction following for complex tasks and seek a way to bootstrap tool-use capabilities without massive human-annotated datasets. START addresses this by using a self-taught approach to bridge the gap between verbal reasoning and algorithmic execution.

Methodology: The START Framework

The researchers propose a multi-stage approach to activate and refine tool-use capabilities, moving from inference-time prompting to permanent model fine-tuning.

The START framework overview, illustrating Hint-infer, Data Selection/SFT, and Rejection Fine-Tuning (RFT).

1. Hint-infer

This is a training-free inference technique. The authors observed that models like QwQ-32B already have a latent ability to use code but don't "know" when to trigger it. By intercepting the model's output at natural pause points (like the word "Wait") and injecting a semantic hint—e.g., "Wait, I can use Python to check if my approach is correct..."—they force the model to switch into a tool-using mode. This allows for "test-time scaling," where more rounds of hint-driven refinement lead to better answers.

2. Hint-RFT (Hint-driven Rejection Fine-Tuning)

To turn this inference trick into a permanent skill, they used Hint-infer to synthesize a high-quality training dataset. Crucially, they employed an active learning strategy: they only kept samples where the injected hint led the model to fix a previously incorrect answer. This ensures the model learns the utility of the tool for error correction.

3. Diverse Hint Patterns

Rather than a single prompt, they used six types of hints to encourage diverse problem-solving strategies, as shown in the Hint-Library.

The Hint-Library containing various functional hints for code and math problems.

Experimental Setup and Results

The authors curated a training set of approximately 50,000 problems across mathematics (AIME, MATH, Numina-MATH) and programming (Codeforces, LiveCodeBench). Evaluation was conducted on rigorous benchmarks including AMC23, AIME24, AIME25, MATH500, GPQA, and LiveCodeBench.

START significantly outperformed its base models (QwQ-32B-Preview and DeepSeek-R1-Distill-Qwen-32B):

  • Competition Math: Achieved 95.0% on AMC23 and 75.6% on AIME24, placing it among the top-performing reasoning models globally.
  • Science: On the PhD-level GPQA benchmark, it reached 64.6%, a 5.5% absolute improvement over the base model.
  • Coding: On LiveCodeBench, the model showed a massive jump in "Medium" difficulty problems (from 46.0% to 84.6%), proving that the self-debugging logic learned in math carries over to pure programming tasks.

Key Insights

The "Efficiency Paradox"

Integrating tools usually adds computational overhead. However, START models actually became more efficient. By replacing verbose, step-by-step manual calculations with a few lines of Python, the total token count dropped significantly (e.g., from ~5,500 tokens to ~3,500 on AMC23).

Emergent Generalization

The model spontaneously began using specialized Python libraries it was never explicitly trained on. For instance, on the GPQA chemistry benchmark, the model invoked rdkit (a cheminformatics library) over 250 times to solve molecular problems, despite rdkit being absent from the training data. This suggests the model learned a general principle of tool-seeking.

Conclusion

While START demonstrates significant gains, limitations remain. Currently, hints are injected at somewhat arbitrary discourse markers. Future work should focus on adaptive strategies where the model learns to decide exactly when it needs a tool based on the complexity of the current reasoning step. Additionally, while math and science saw huge gains, the improvement on "Hard" coding problems was modest (+2.0%), suggesting that deeply architectural software engineering tasks remain a challenge.

Summary updated

BibTeX
@misc{li2025start,
  author = {Cheng‐Peng Li and Mingfeng Xue and Zhenru Zhang and Jiaxi Yang and Beichen Zhang and Bowen Yu and Binyuan Hui and Junyang Lin and Xiang Wang and Dayiheng Liu},
  title = {{START: Self-taught Reasoner with Tools}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.683},
}
Figure