Large Language Models (LLMs) possess a remarkable ability to solve complex problems by thinking out loud via Chain-of-Thought (CoT) prompting. However, deploying these hundred-billion-parameter giants is often impractical due to high latency and computational costs.
Traditional Knowledge Distillation (KD) offers a classic solution: train a smaller "student" model to mimic the output probabilities (logits) or final answers of the larger "teacher." Yet, when applied to complex reasoning tasks, standard distillation struggles. It forces the student to guess the underlying logic behind an answer based solely on the final output.
Rationale distillation flips this paradigm. Instead of just teaching the student what the right answer is, the teacher transfers how it arrived there by distilling its intermediate reasoning steps.
The Limits of Mimicking Answers
Why does standard label-based distillation fail for complex tasks? When we train a model only on final answers, we are optimizing within its existing knowledge base. As Yue et al. observe, methods like reinforcement learning or standard supervised fine-tuning can narrow a model's reasoning boundary, making it highly efficient at sampling answers it already understands but failing to introduce fundamentally new cognitive patterns.
To expand this reasoning boundary, a student model needs explicit exposure to structured logic. By training the student to replicate the intermediate steps generated by a teacher, the student internalizes the underlying problem-solving process. This approach allows smaller models to inherit complex capabilities—such as symbolic deduction or multi-hop inference—that were previously thought to be emergent properties exclusive to massive scales (Fang et al.).
The Multi-Task Mechanics of Rationale Distillation
In practice, teaching a model to reason requires structuring the training objective so that the logic directly informs the final prediction. A foundational framework for this is proposed by Hsieh et al., who frame rationale distillation as a multi-task learning problem.
Instead of a single objective, the student model is optimized using a dual-headed or multi-task loss function:
- The Rationale Generation Task: The student learns to generate the natural language explanation provided by the teacher.
- The Label Prediction Task: The student learns to predict the correct final answer or classification.
Input Text ───► [ Student Model ] ───┬─► Predicted Rationale (Why)
└─► Predicted Label (What)
This dual objective fundamentally changes how the student updates its weights. Rather than finding spurious correlations between the raw input and the final label, the student is forced to construct a causal pathway. By treating rationale generation as a secondary task during training, the student learns the conceptual framework required for the primary task.
Generating High-Fidelity Supervision Chains
A student is only as good as its teacher's lessons. If a teacher model generates a flawed or hallucinated rationale, the student will internalize incorrect logic. To ensure a clean training signal, researchers use specific conditioning strategies to generate high-quality rationales.
One prominent technique is oracle conditioning. When generating the dataset of rationales, the teacher model is explicitly provided with the ground-truth answer. For instance, Liu et al. and Joo et al. condition the teacher's generation phase on known correct answers. This forces the teacher to backward-reason from a correct conclusion, ensuring that the intermediate steps are logically sound and tightly coupled with the truth.
To further tighten this bond, Chen et al. demonstrate the importance of maximizing the mutual information between the generated rationales and the final answers. This prevents a common failure mode where a student model learns to output superficial, boilerplate explanations that look correct but are completely decoupled from its actual final prediction.
Bypassing the Inference Tax
While generating step-by-step explanations makes a model more accurate, it introduces a severe practical bottleneck: the inference tax. Generating dozens of intermediate reasoning tokens at runtime dramatically slows down inference speeds and increases token costs.
Can we get the accuracy benefits of reasoning without paying the token tax at test time? Recent work shows that the reasoning process can be internalized or bypassed entirely during deployment.
Train-Time-Only Reasoning
Xu introduced Dual-Head Reasoning Distillation (DHRD), a method that uses an auxiliary reasoning head only during training. The teacher's rationales are used to supervise the internal state development of the student, ensuring its representations align with logical causal chains. At test time, the reasoning head is discarded, allowing the model to output direct classifications up to 142× faster than standard CoT methods while retaining the accuracy boost.
Latent Space Shortcuts
Rather than discarding the reasoning path, models can learn to compress it. Wang et al. developed a framework that decouples the acquisition of reasoning from computational overhead. The student first aligns its internal latent representations with the explicit text-based reasoning path of the teacher. Once this alignment is achieved, the model is trained to take dynamic shortcuts in its hidden layers, skipping non-critical steps and accelerating inference by over 20× without losing the underlying logical depth.
Why Language is a Superior Supervisor
Traditional knowledge distillation often focuses on feature matching—forcing the student's hidden layers to match the mathematical geometry of the teacher's layers. However, this is highly restrictive when the student and teacher have completely different architectures (e.g., distilling a 175B parameter dense model into a 7B parameter model).
Language acts as a universal, low-dimensional interface for high-dimensional logic. As Wu demonstrates, transferring the explicit reasoning behind a prediction—such as token-level attribution scores or natural language rationales—improves student performance more effectively than matching internal vector representations.
By using language as the medium of distillation, the student model is not forced to mimic the exact neural firing patterns of a larger architecture. Instead, it learns a highly generalizable, human-readable causal framework of the domain, making rationale distillation one of the most effective ways to build compact, highly capable, and explainable models.
Key Foundational Work
HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning (2023) — contributed 2 concept elements to this direction.
Large Language Models Are Reasoning Teachers (2023) — contributed 1 concept elements to this direction.
The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning (2023) — contributed 1 concept elements to this direction.