Distilling Reasoning Trajectories into Compact Language Models

665 concepts482 papers10 proposals

When faced with complex tasks like multi-step mathematics, symbolic planning, or open-world tool use, large language models (LLMs) with hundreds of billions of parameters excel by generating intermediate "chain-of-thought" rationales. Conversely, small language models (sLMs) typically fail at these tasks. They lack the internal parameter capacity to store vast amounts of world knowledge or execute complex, implicit multi-step computations in a single forward pass.

A powerful paradigm has emerged to bridge this gap: reasoning trajectory distillation. Instead of training smaller models to mimic only the final outputs of larger teacher models, researchers fine-tune them on the complete intermediate cognitive paths—including rationales, sub-goals, tool calls, and self-corrections—generated by state-of-the-art architectures.

This approach shifts the burden of performance from raw parameter scale to structured execution. By learning how to think rather than just what to think, compact models can achieve performance levels previously thought to be exclusive emergent properties of massive scale.


The Core Intuition: Decomposing the Function Space

To understand why reasoning distillation works, consider the task of mapping a complex question $Q$ to a final answer $A$.

Directly mapping $Q \to A$ requires the model to represent a highly non-linear, discontinuous function within its weights. A massive model has the representational capacity to approximate this function implicitly during its forward pass. A small model does not; it trips over the logical leaps, resulting in hallucinations or structural failures.

Direct Mapping (Hard for Small Models):
[Complex Question Q] ───────────────────────────────────────────► [Final Answer A]

Trajectory Mapping (Easier for Small Models):
[Complex Question Q] ──► [Step 1] ──► [Step 2] ──► [Step 3] ──► [Final Answer A]

Reasoning trajectory distillation breaks this massive functional leap into a sequence of localized, highly predictable transitions:

$$Q \to T_1 \to T_2 \to \dots \to T_n \to A$$

where $T_i$ represents intermediate thoughts or actions. For a small model, predicting the next logical token in a step-by-step rationale is a significantly simpler optimization problem than generating the final answer directly. Each intermediate step is emitted into the context window, acting as an external "scratchpad." This dynamically offloads working memory and allows the model to condition its next computation on its own explicit, intermediate progress.

As demonstrated by Lucie Charlotte Magister et al. in Teaching Small Language Models to Reason, fine-tuning smaller architectures on these explicit chains allows a 220M-parameter model to match or exceed the performance of much larger baselines. In a systematic review, Zheng Chu et al. in Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future confirm that the ability to generate step-by-step rationales is not an exclusive artifact of model size, but a highly transferable skill that can be internalized via targeted supervised fine-tuning.


Agentic Distillation: Offloading Capacity to Tools

The strategy becomes even more potent when the reasoning trajectories involve external interactions. Small models are fundamentally limited by their parameter budget; they cannot memorize every obscure fact or perform flawless mental arithmetic.

Agentic trajectory distillation solves this by training the student model to act as a coordinator rather than an encyclopedia. The teacher model generates trajectories that interleave internal reasoning ("thoughts") with external actions ("calls" to search engines, code interpreters, or databases), a paradigm popularized by Shunyu Yao et al. in ReAct: Synergizing Reasoning and Acting in Language Models.

[Thought]       "I need to calculate the compound interest for $5000."
[Action]        [Call Code Interpreter: `5000 * (1 + 0.05)**10`]
[Observation]   "8144.47"
[Thought]       "The final amount is 8144.47. Now I can formulate the answer."

When an sLM is fine-tuned on these interactive sequences, it learns the precise syntax and strategic timing required to invoke external tools. As Minki Kang et al. show in Distilling LLM Agent into Small Models with Retrieval and Code Tools, this allows the student to offload heavy computational and memory burdens to the environment. The student model doesn't need the capacity to compute the math or store the fact; it only needs the capacity to generate the correct API call sequence based on the context.


Structuring the Trajectory

While natural language chains-of-thought are highly expressive, they can also be noisy, verbose, and ambiguous for a small model to emulate perfectly. To optimize the transfer of logic, recent techniques focus on structuring the distilled trajectories.

1. Sub-goal Decomposition

Instead of letting the model wander through a loose narrative, the teacher's trajectory is structured into explicit, high-level sub-goals. Elias Stengel-Eskin et al. demonstrate this in Sub-goal Distillation: A Method to Improve Small Language Agents, where a teacher model annotates expert trajectories with programmatic sub-goals. This creates a hierarchical execution plan that a compact model can easily follow and execute step-by-step.

2. Code-Based Rationales

An alternative to natural language explanation is distilling logic directly into code format (such as Python or formal logic). Zhanming Jie et al. explore this in Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning, finding that fine-tuning smaller models on structured, code-based reasoning chains yields higher logical precision than training on unstructured text. Code provides a rigid, unambiguous syntax that acts as a cleaner optimization signal during gradient descent.


Why It Works: Dense Supervision Signals

Traditional knowledge distillation minimizes the cross-entropy loss between the student's output and the teacher's final answer token. This provides a sparse learning signal: the student is punished or rewarded based solely on whether it arrived at the correct destination, receiving no guidance on the correctness of its internal path.

Reasoning trajectory distillation provides a dense supervision signal. Because the training data includes every intermediate token of the reasoning path, the model is supervised at every single step of its cognitive journey.

Standard Distillation (Sparse Signal):
Question ──────────────────────────────────────────────────► Answer (Loss calculated here)

Trajectory Distillation (Dense Signal):
Question ──► Thought 1 ──► Thought 2 ──► Tool Call ──► Observation ──► Answer
               ▲            ▲             ▲             ▲               ▲
         (Loss)       (Loss)        (Loss)        (Loss)          (Loss)

This dense supervision forces the student model to align its internal attention distributions and hidden representations with the structured problem-solving methodologies of the teacher. As argued by Yao Fu et al. in Specializing Smaller Language Models towards Multi-Step Reasoning, by narrowing a small model's focus away from open-ended generation and toward these dense, multi-step reasoning paths, the model's limited capacity becomes highly specialized, unlocking performance that rivals models an order of magnitude larger.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reasoning_distillation_12f6630f,
  title = {reasoning distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reasoning-distillation/rdr_12f6630f50b647e18a530f60ef4026d6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reasoning_distillation_12f6630f_v1,
  title = {reasoning distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reasoning-distillation/rdr_12f6630f50b647e18a530f60ef4026d6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake