Abstract
Large language models (LLMs) often struggle with mathematical problems that require exact computation or multi-step algebraic reasoning. Tool-integrated reasoning (TIR) offers a promising solution by leveraging external tools such as code interpreters to ensure correctness, but it introduces inference-time dependencies that hinder scalability and deployment. In this work, we propose a new paradigm for distilling tool knowledge into LLMs purely through natural language. We first construct a Solver Agent that solves math problems by interleaving planning, symbolic tool calls, and reflective reasoning. Then, using a back-translation pipeline powered by multiple LLM-based agents, we convert interleaved TIR traces into natural language reasoning traces. A Translator Agent generates explanations for individual tool calls, while a Rephrase Agent merges them into a fluent and globally coherent narrative. Empirically, we show that fine-tuning a small open-source model on these synthesized traces enables it to internalize both tool knowledge and structured reasoning patterns, yielding gains on competition-level math benchmarks without requiring tool access at inference.
Introduction
Large Language Models (LLMs) are notoriously shaky when it comes to "System 2" tasks like exact arithmetic, multi-step algebraic manipulation, or symbolic calculus. To address this, many developers employ Tool-Integrated Reasoning (TIR), where the model calls an external code interpreter to handle complex computations. However, TIR introduces significant deployment friction, requiring secure sandboxed environments and increasing latency. This research investigates whether a model can internalize the competence of a tool-user without actually requiring the tool at inference time.
Methodology
The researchers propose a new paradigm for knowledge distillation using a multi-agent back-translation pipeline. The process follows three main stages:
1. Generating TIR Traces
A Solver Agent (powered by GPT-4o-mini) solves math problems by interleaving natural language planning with calls to a curated SymPy toolkit. This creates a "trace" that follows a structured pattern: Plan → Tool Call → Result → Reflection → Next Step.
2. Back-Translation Pipeline
Since raw tool calls (JSON/code) are not ideal for standard language model training, three specialized agents are used to "translate" them into natural language:
- Translator Agent: Converts abstract tool calls into step-by-step natural language explanations.
- Judge Agent: Verifies that the natural language explanation reaches the same mathematical result as the original tool.
- Rephrase Agent: Merges the planning, explanations, and results into a single, fluent, and globally coherent narrative.
3. Supervised Fine-Tuning (SFT)
The team fine-tuned a Qwen2.5-MATH-7B-Instruct model on 11.6k of these high-quality, back-translated traces. These traces were synthesized from a pool of 25k problems sourced from NuminaMATH, MATH, and DeepScaleR.
Key Results and Insights
The fine-tuned model showed impressive gains on difficult benchmarks compared to the base instruct model, despite having no tool access during evaluation:
- AIME: Improved from 7.8% to 10.0%.
- Olympiad Bench: Improved from 41.2% to 43.1%.
- AMC: Improved from 51.2% to 52.4%.
The "Over-Complication" Tax
Interestingly, while the model became much better at "hard" math, it actually performed slightly worse on simpler benchmarks like GSM8K (dropping from 95.5% to 91.1%). The researchers found that the model began to "over-reason," applying complex algebraic variables to simple word problems that could have been solved with basic arithmetic. This suggests that the model successfully internalized the structured reasoning patterns of the symbolic tools, even when they were unnecessary.
Conclusion and Future Work
This work demonstrates that structured, high-fidelity reasoning data is more important than raw volume for mathematical distillation. By reading back-translated traces, models learn to decompose problems and verify steps like a symbolic solver. Future work will focus on expanding tool coverage beyond SymPy and exploring "hybrid" training—mixing detailed traces with concise Chain-of-Thought data—to prevent over-complication on simple tasks while retaining advanced reasoning capabilities.
Summary updated