Distilling Tool Knowledge into Language Models via Back-Translated Traces

Xingyue Huang, Xianglong Hu, Zifeng Ding, Yuan He, Rishabh, Waleed Alzarooni, Ziyu Ye, Wendong Fan, Bailan He, Haige Bo, Changran Hu, Guohao LiarXiv 2025 · OpenReview

Abstract

Large language models (LLMs) often struggle with mathematical problems that require exact computation or multi-step algebraic reasoning. Tool-integrated reasoning (TIR) offers a promising solution by leveraging external tools such as code interpreters to ensure correctness, but it introduces inference-time dependencies that hinder scalability and deployment. In this work, we propose a new paradigm for distilling tool knowledge into LLMs purely through natural language. We first construct a Solver Agent that solves math problems by interleaving planning, symbolic tool calls, and reflective reasoning. Then, using a back-translation pipeline powered by multiple LLM-based agents, we convert interleaved TIR traces into natural language reasoning traces. A Translator Agent generates explanations for individual tool calls, while a Rephrase Agent merges them into a fluent and globally coherent narrative. Empirically, we show that fine-tuning a small open-source model on these synthesized traces enables it to internalize both tool knowledge and structured reasoning patterns, yielding gains on competition-level math benchmarks without requiring tool access at inference.

Introduction

Large Language Models (LLMs) are notoriously shaky when it comes to "System 2" tasks like exact arithmetic, multi-step algebraic manipulation, or symbolic calculus. To address this, many developers employ Tool-Integrated Reasoning (TIR), where the model calls an external code interpreter to handle complex computations. However, TIR introduces significant deployment friction, requiring secure sandboxed environments and increasing latency. This research investigates whether a model can internalize the competence of a tool-user without actually requiring the tool at inference time.

Methodology

The researchers propose a new paradigm for knowledge distillation using a multi-agent back-translation pipeline. The process follows three main stages:

1. Generating TIR Traces

A Solver Agent (powered by GPT-4o-mini) solves math problems by interleaving natural language planning with calls to a curated SymPy toolkit. This creates a "trace" that follows a structured pattern: Plan → Tool Call → Result → Reflection → Next Step.

Example of a high-level plan generated by the Solver Agent for a geometry problem.

2. Back-Translation Pipeline

Since raw tool calls (JSON/code) are not ideal for standard language model training, three specialized agents are used to "translate" them into natural language:

  • Translator Agent: Converts abstract tool calls into step-by-step natural language explanations.
  • Judge Agent: Verifies that the natural language explanation reaches the same mathematical result as the original tool.
  • Rephrase Agent: Merges the planning, explanations, and results into a single, fluent, and globally coherent narrative.
Example of a step execution trace involving a symbolic tool call (SymPy) and its result.

3. Supervised Fine-Tuning (SFT)

The team fine-tuned a Qwen2.5-MATH-7B-Instruct model on 11.6k of these high-quality, back-translated traces. These traces were synthesized from a pool of 25k problems sourced from NuminaMATH, MATH, and DeepScaleR.

Key Results and Insights

The fine-tuned model showed impressive gains on difficult benchmarks compared to the base instruct model, despite having no tool access during evaluation:

  • AIME: Improved from 7.8% to 10.0%.
  • Olympiad Bench: Improved from 41.2% to 43.1%.
  • AMC: Improved from 51.2% to 52.4%.

The "Over-Complication" Tax

Interestingly, while the model became much better at "hard" math, it actually performed slightly worse on simpler benchmarks like GSM8K (dropping from 95.5% to 91.1%). The researchers found that the model began to "over-reason," applying complex algebraic variables to simple word problems that could have been solved with basic arithmetic. This suggests that the model successfully internalized the structured reasoning patterns of the symbolic tools, even when they were unnecessary.

Conclusion and Future Work

This work demonstrates that structured, high-fidelity reasoning data is more important than raw volume for mathematical distillation. By reading back-translated traces, models learn to decompose problems and verify steps like a symbolic solver. Future work will focus on expanding tool coverage beyond SymPy and exploring "hybrid" training—mixing detailed traces with concise Chain-of-Thought data—to prevent over-complication on simple tasks while retaining advanced reasoning capabilities.

Summary updated

Abstract

Large language models (LLMs) often struggle with mathematical problems that require exact computation or multi-step algebraic reasoning. Tool-integrated reasoning (TIR) offers a promising solution by leveraging external tools such as code interpreters to ensure correctness, but it introduces inference-time dependencies that hinder scalability and deployment. In this work, we propose a new paradigm for distilling tool knowledge into LLMs purely through natural language. We first construct a Solver Agent that solves math problems by interleaving planning, symbolic tool calls, and reflective reasoning. Then, using a back-translation pipeline powered by multiple LLM-based agents, we convert interleaved TIR traces into natural language reasoning traces. A Translator Agent generates explanations for individual tool calls, while a Rephrase Agent merges them into a fluent and globally coherent narrative. Empirically, we show that fine-tuning a small open-source model on these synthesized traces enables it to internalize both tool knowledge and structured reasoning patterns, yielding gains on competition-level math benchmarks without requiring tool access at inference.
BibTeX
@misc{huang2025distilling,
  author = {Xingyue Huang and Xianglong Hu and Zifeng Ding and Yuan He and Rishabh and Waleed Alzarooni and Ziyu Ye and Wendong Fan and Bailan He and Haige Bo and Changran Hu and Guohao Li},
  title = {{Distilling Tool Knowledge into Language Models via Back-Translated Traces}},
  year = {2025},
  eprint = {2506.19171},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.19171},
}
Figure