A common way to build a small, cheap text classifier is to have a large language model write explanations and then train the small model on them. The large "teacher" model writes a short explanation, or rationale, for each training example. The small "student" model is then trained to predict both the label and that rationale, and at test time it outputs only the label. Distilling Step-by-Step showed this works strikingly well: a 220M-parameter T5 student beat its 540B PaLM teacher on e-SNLI and matched standard fine-tuning with 12.5% of the labeled data. The rationales in that pipeline, and in RB-FT, come from a handful of hand-written prompts. Nobody has asked which teacher prompt produces rationales the student actually learns best from. Nobody has asked what those rationales cost to train on, either: every extra rationale token is an extra token of backpropagation in every training epoch.
We propose to choose the teacher's prompt automatically. Each candidate prompt is scored by how well a small student learns from the rationales it produces, per token of training it costs. The score does not use how accurate the teacher's own answers are, and it does not use how plausible the rationales look to people. The bet is that "a prompt that makes the teacher accurate" and "a prompt whose rationales teach a small model well and cheaply" are different targets. If they are different, putting the student inside the prompt-search loop should buy accuracy and training savings that standard prompt optimizers cannot. If they turn out to be the same, the experiments will show that too, and the answer is still useful.
Core Claim
Two tensions in the literature suggest that the prompts we use to elicit rationales are tuned for the wrong target. The first is about length and detail. Explain Yourself! (CAGE) and RB-FT treat dense, descriptive rationales as better supervision. Verbosity-Aware Rationale Reduction (VARR) instead finds that removing redundant sentences cut generated tokens by 19.87% while raising accuracy by 7.71% on average. Post-Semantic-Thinking shows that errors in reasoning written before the answer spill into the label. The second tension is about what prompt optimizers reward. CAPO and ZOPO search for prompts that maximize the prompted model's own accuracy. That is the right objective when the prompt is used at deployment time. It is not obviously right when the prompt's only job is to produce training data for a different, much smaller model.
Our claim is that one can search the space of teacher prompts against a cheap, student-in-the-loop estimate of distillation value, and that doing so beats both hand-written prompts and teacher-accuracy-optimized prompts. It should give a better accuracy-per-training-cost trade-off. We expect it to give higher accuracy in low-data regimes and shorter rationales, which lowers memory and GPU-hours in the student's multi-task training. Formally, for a candidate teacher prompt $p$, let $\theta_p$ be a small proxy student briefly fine-tuned on a small labeled subset annotated with $p$'s rationales. Let $\mathrm{Acc}_{\mathrm{val}}(\theta_p)$ be its held-out label accuracy, and let $\bar{L}(p)$ be the mean rationale length in tokens. We maximize
$$J(p) = \mathrm{Acc}_{\mathrm{val}}(\theta_p) - \lambda \, \bar{L}(p),$$
where $\lambda$ sets the exchange rate between accuracy and training tokens. We sweep $\lambda$ to trace a curve of accuracy against cost rather than report a single point.
Relation to Existing Work
The closest system is DSPy. DSPy compiles pipelines by bootstrapping demonstrations that pass a task metric, and it can distill a compiled program into a 770M T5 (39.3% exact match on HotPotQA). Its optimizer keeps demonstrations and instructions according to how the prompted pipeline scores. The fine-tuned model is a downstream consumer, not the judge. We make the student the judge.
CAPO supplies most of our search machinery. We reuse its evolutionary search over instructions and examples, its statistical racing (paired t-tests over evaluation blocks, which it reports saves 44% of evaluations), and its length penalty. We change what an "evaluation" is: instead of scoring the teacher's answers, each block is a short student training run.
VARR constrains our design as well as motivating it. It shows verbosity is real, but it removes verbosity afterwards through repeated likelihood evaluations during training. We try to avoid producing the verbosity in the first place.
S2LPP is the reason we think a small proxy student can stand in for the real one. It found that prompt preferences stay fairly consistent across model sizes, with recovery rates often above 70%. We test whether that consistency also holds when the "preference" is distillation value rather than zero-shot accuracy.
Knowledge Injection via Prompt Distillation also distills a prompted teacher into a LoRA student. Its teacher context is fixed and hand-designed, and its target is factual knowledge, not rationales.
Several papers mention zeroth-order and frequency-domain fine-tuning methods (KerZOO, MaCP) as ways to cut fine-tuning memory. We treat them as compatible with our method but not central to it. Our lever on fine-tuning cost is the length and learnability of the training sequences, which helps any optimizer.
Research Program
The pipeline has three parts.
First, a pool of candidate teacher prompts. We seed it with the Distilling Step-by-Step few-shot chain-of-thought prompts, the four-aspect structured prompt style of RB-FT, a "post-thinking" (answer first, then justify) format from Post-Semantic-Thinking, and plain "Let's think step by step." CAPO-style mutation and crossover, run by the teacher model itself, then expand this pool.
Second, the proxy evaluator. The teacher annotates a fixed subset of about 500 labeled examples under a candidate prompt. We then fine-tune a small student, T5-base (220M), with LoRA for a fixed small number of steps on the multi-task label-plus-rationale objective and measure validation accuracy. CAPO's racing enlarges the subset only for candidates that survive the t-tests. Teacher generations are cached, so the cost of one evaluation is dominated by the short student training run.
Third, the full distillation. We apply the chosen prompt to the full training set and distill into T5-base, T5-large (770M), and a LoRA-adapted Llama 3.2 1B. The Llama student tests whether the choice made with a T5 proxy transfers across model families. The teacher throughout is an open model, Llama-3.3-70B, which CAPO also used, so that all generation costs can be counted.
The first analysis we run, before any claims about gains, is a calibration study. We sample about 30 prompts that span the pool. For each, we compute three things: teacher accuracy on the training labels, the proxy score, and full-distillation student accuracy. We then report the Spearman rank correlations among them. This single study tells us whether the method has a reason to exist, as discussed below.
Baselines
The naive reference point is label-only fine-tuning: the same students, the same LoRA configuration, the same labeled-data fractions, trained with no rationales. It fixes what rationales add at all. RB-FT reports that on SmartHome-LLM, direct supervised fine-tuning reached 76.02% against 82.65% with rationale bootstrapping, so we expect a clear gap, but we measure it on our own tasks.
The main existing method is Distilling Step-by-Step with its original hand-written prompts, run with the same teacher, students, data fractions, training steps, and learning-rate search budget as our method. It is the strongest competitor because it already produces most of the gain from rationales and is the published pipeline we would replace. On e-SNLI the file reports only relative results for it: the student outperformed the 540B teacher and matched full fine-tuning with 12.5% of the data. No reported accuracy value was found on Lacuna for Distilling Step-by-Step on e-SNLI, ANLI, or CommonsenseQA, so we reproduce it ourselves rather than quote a number. Beating it means higher student test accuracy at 12.5% and 25% data with equal or fewer training tokens. It also means the improvement holds after we charge our method for its prompt-search compute.
The strongest prompt-optimization competitor is CAPO optimizing teacher accuracy. This is the same search algorithm, candidate pool, racing procedure, and number of teacher calls as ours, but each candidate is scored by the teacher's own labeled accuracy with CAPO's length penalty. The reported strength of CAPO is outperforming other discrete optimizers in 11 of 15 cases and gaining up to 21 percentage points on GSM8K with Llama-3.3-70B. That result concerns the prompted model's accuracy; no reported result on distillation was found on Lacuna for CAPO. This baseline is also the key ablation. Everything is identical except whether the student sits inside the scoring loop, so any gap isolates the core ingredient.
Three further ablations complete the comparison. The first sets $\lambda = 0$ to separate the effect of the length term from the effect of student-based scoring. The second picks a random prompt from the evolved pool, which checks that search helps at all. The third applies VARR-style sentence pruning after generation to the hand-written-prompt rationales, with its training-time overhead counted in the cost.
Why not just…?
Why not just keep the hand-written chain-of-thought prompt? It already produces the headline Distilling Step-by-Step results, so this is a fair challenge. The evidence against treating hand-written prompts as final comes from other settings. ZOPO's discovered prompt lifted GSM8K from 71.80% to 75.36% over "Let's think step by step," and DSPy's compilation raised Llama2-13b-chat on GSM8K from 9.4% to 46.9%. Neither result measures distillation, though. Our head-to-head comparison at matched data fractions settles the question. If the gain over the hand-written prompt is within seed noise on e-SNLI, ANLI, and CommonsenseQA, we will say so.
Why not just optimize the teacher prompt for teacher accuracy with an existing optimizer? This is the real threat. If a teacher that is right more often also writes rationales that teach better, CAPO or DSPy already solve the problem, and a student in the loop only adds cost. The calibration study answers it directly. If teacher accuracy correlates with full-distillation accuracy about as well as our proxy does, and the CAPO-teacher-accuracy baseline matches our method, then the conclusion is that existing optimizers suffice. We would report that and drop the claim. We expect them to diverge for two reasons. Prompts that improve teacher accuracy through long, exploratory reasoning inflate training sequences. And post-thinking formats can leave teacher accuracy unchanged while changing what the student learns, as Post-Semantic-Thinking suggests.
Why not just filter or prune rationales after generating them? Keeping only rationales whose teacher label is correct, or pruning sentences as VARR does, are cheap alternatives. Label-correctness filtering is applied in all conditions, including ours, so it cannot explain our gains. VARR pruning is run as a baseline. The cost of its repeated likelihood evaluations during training is charged against it, and our method is also tested with pruning on top. If pruning alone closes the gap, the prompt search is not worth its cost.
Why not just skip fine-tuning and prompt the big model at inference time? For classification served at volume, repeated inference cost dominates. Inference-Time Distillation reaches teacher-level results on agent tasks through retrieval, but it reports that coverage gaps in the retrieval database cause failures. We include the optimized teacher prompt used directly at inference as a reference row, with per-example token cost reported next to accuracy.
Proposed Experiments
The benchmarks are e-SNLI, ANLI, and CommonsenseQA with CoS-E explanations. These are the Distilling Step-by-Step tasks. MNLI and HANS serve as out-of-distribution checks for students trained on e-SNLI, which follows the robustness framing of the span-level NLI work. All conditions run at 12.5%, 25%, and 100% of the labeled data, with three seeds each.
We report student test accuracy together with a full cost account. The account covers teacher tokens generated during search and annotation, student proxy-training GPU-hours during search, and final student training GPU-hours and peak memory. It also includes the break-even point: the number of downstream distillation runs, or model sizes, over which the search cost pays for itself. As a secondary check that shorter rationales have not become meaningless, we score rationale overlap with the human e-SNLI and CoS-E explanations. We also record how often rationales merely repeat the input, the "vacuous repetition" failure noted in the original e-SNLI work.
Three results are decisive. First, in the calibration study, the proxy score must rank prompts by full-distillation accuracy substantially better than teacher accuracy does. Otherwise the core premise fails. Second, the chosen prompt must beat both the hand-written and the CAPO-teacher-accuracy baselines at 12.5% data on at least two of the three tasks, while using no more training tokens. Third, a prompt chosen with the T5-base proxy must keep most of its advantage when applied to T5-large and to the Llama 3.2 1B student. If the gain appears only for the proxy model, the method is overfitting a single student rather than finding transferable rationale styles. That would limit the claim to per-student search, which is useful only when the student is fixed in advance.
References
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification
- Explain Yourself! Leveraging Language Models for Commonsense Reasoning
- Verbosity-Aware Rationale Reduction: Sentence-Level Rationale Reduction for Efficient and Effective Reasoning
- Distilling Reasoning Ability From Large Language Models With Adaptive Thinking
- CAPO: Cost-Aware Prompt Optimization
- Localized Zeroth-Order Prompt Optimization
- DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines
- S2LPP: Small-to-Large Prompt Prediction across LLMs
- Knowledge Injection via Prompt Distillation
- KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuning
- MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
- Inference-Time Distillation: Cost-Efficient Agents Without Fine-Tuning or Manual Prompt Engineering
- Logical Reasoning with Span Predictions: Span-level Logical Atoms for Interpretable and Robust NLI Models
- e-SNLI: Natural Language Inference with Natural Language Explanations