Imputing Teacher Reasoning in Latent Space to Cut the Rationale Budget

Training a small model on a large model's written explanations makes it a much better classifier. The cost is that someone has to pay the large model to write an explanation for every training example, and then the small model has to be trained on all those long token sequences. This note proposes paying for explanations on only a small seed fraction of the data. On that seed, we learn how the student's internal representation shifts when it reads a good explanation, both the average shift and its spread. For the remaining label-only examples, we then synthesize "reasoned" versions of their representations directly in hidden space. No new tokens are generated, filtered, or trained on. The question is how many teacher explanations such latent imputation can replace before accuracy degrades. We also ask whether the benefit comes from reasoning-shaped structure or merely from the generic noise regularization already known from contrastive recommendation.

The opportunity sits between three bodies of work. Rationale distillation is very data-efficient per labeled example but expensive per rationale. Synthetic-data pipelines spend most of their compute on generation and rejection. Embedding-noise augmentation, the "topology-preserving" family that perturbs hidden vectors instead of editing graphs or text, shows that cheap continuous views can beat costly discrete ones. Nobody has asked whether that third idea can stand in for the second in the setting of the first. We also ask it in a way that can come out negative.

Core Claim

In train-time-only rationale distillation, where the explanation shapes the student's representation during training and is discarded at test time, most of the benefit can be kept with teacher rationales on 10–25% of the data. The student is trained on the rest by mapping each question-only representation to a predicted "after-reading-the-rationale" representation. Views are then drawn around that prediction with noise whose covariance matches the residual spread observed on the seed set. The hypothesis has two parts. First, this direction-aware latent imputation beats training the unannotated majority on labels alone. Second, it beats isotropic noise of the same magnitude. If the second part fails, the gain is plain uniformity regularization of the kind identified in Are Graph Augmentations Necessary? (SimGCL), not imputed reasoning, and we will report it that way.

Research Program

The base system is the dual-head design of Dual-Head Reasoning Distillation (DHRD). A decoder backbone has two heads. A classification head reads a pooled representation. An auxiliary next-token head learns to reproduce teacher rationales during training and is dropped at inference, giving 96–142× higher throughput than chain-of-thought decoding. DHRD supplies the method. Its own shuffling ablation, where scrambled rationales collapse accuracy, motivates us: the representation change caused by a coherent rationale carries real signal that might be learnable.

On the seed set $S$, where teacher rationales exist, we compute two pooled student states for each example. The first, $h_q$, comes from reading the question alone. The second, $h_{qr}$, comes from reading the question followed by the teacher rationale. We train these with the DHRD losses and a classification loss on both states. We also fit a small network $g$ to predict $h_{qr}$ from $h_q$. The residuals $h_{qr} - g(h_q)$ are collected, and their covariance is summarized by its top-$k$ principal directions $U_k$ with variances $s^2$. This mirrors how COSTA builds augmented node features that preserve second-order statistics rather than adding arbitrary noise. We borrow its covariance-preservation idea but estimate the covariance from teacher-induced shifts rather than from raw features.

For an unannotated example with label $y$, we draw a view $\tilde h = g(h_q) + U_k\,\mathrm{diag}(s)\,z$ with $z$ standard Gaussian. We then minimize

$$ \mathcal{L} = \mathrm{CE}\big(f(h_q), y\big) + \lambda_1\,\mathrm{CE}\big(f(\tilde h), y\big) + \lambda_2\,\mathrm{InfoNCE}\big(h_q, \tilde h\big), $$

where $f$ is the classification head and InfoNCE is the standard contrastive loss. That loss pulls each question's own imputed view toward its question state and pushes it away from other examples' views in the batch.

Gradients flow through $g(h_q)$ into the backbone. As a result, the question-only representation, which is the only thing used at test time, is shaped so that its "reasoned" neighbourhood is classifiable. This plays the role DHRD's reasoning head plays, but without rationale tokens. We cross-layer contrast as in XSimGCL only if a pilot shows that a single pooled layer is insufficient. Since XSimGCL reports that layer choice must be tuned per dataset, this would be a measured choice, not a default.

Cost accounting is part of the deliverable. For every run we report teacher tokens generated, training tokens processed, and GPU-hours. This places each method on an accuracy-versus-cost curve rather than at a single point.

Baselines

The naive reference is label-only fine-tuning of the same backbone on all examples. Distilling Step-by-Step shows this baseline is weak. Rationale multi-task training matched it with 12.5% of e-SNLI data, and on ANLI a 770M student with rationales beat the 540B teacher at 80% data where standard fine-tuning failed even at 100%. Our method must clearly exceed label-only training at every seed fraction to be worth anything.

The strongest competitor is full DHRD with teacher rationales on 100% of the data. It is the ceiling we are trying to approach cheaply. It is the right target because it is the most inference-efficient rationale method in the file and keeps rationales out of test time, exactly like ours. DHRD reports relative accuracy gains of 0.65% to 5.47% over standard classifiers on SuperGLUE, largest on COPA and CB. The file gives no absolute task accuracies, so we will reproduce DHRD ourselves with identical backbone, teacher, and tuning budget.

"Beating" this baseline means one of two outcomes. The first is matching full DHRD within one standard deviation over five seeds while using at most 25% of its teacher tokens. The second is exceeding the seed-only DHRD baseline at equal teacher-token cost on most tasks. The seed-only baseline uses DHRD on $S$ plus label-only training on the rest, with no imputation. It is the most important comparison, because it holds teacher cost fixed and isolates imputation.

Two ablations isolate the key ingredient. The isotropic ablation replaces $g(h_q)+U_k\mathrm{diag}(s)z$ with $h_q$ plus SimGCL-style bounded uniform noise of matched norm. The shuffled-seed ablation fits $g$ and the covariance on rationales whose sentences were permuted. If either matches the full method, the claim that reasoning structure is being imputed is false. All methods get the same hyperparameter search budget, measured in trials, and the same number of training steps.

Why not just…?

Why not just generate rationales for every example with a cheap open-weight teacher? This is the real threat. ScaleQuest cut the cost of a million math instances to roughly $680 in GPU rental, versus about $6,000 with GPT-4o, by using a 7B generator. If a small open teacher writes adequate rationales for all data, the generation overhead may be too small to matter. Two costs survive that argument. ScaleQuest still generated two million raw instances to keep one million. And training on long rationale sequences multiplies training tokens even when generation is cheap. We settle the question empirically by including "cheap-teacher DHRD on 100%" as an additional point on the cost curve. If it dominates our method in accuracy at equal or lower total GPU-hours, generation plus training, we conclude latent imputation is not worth it.

Why not just add embedding noise to the student, as in SimGCL? SimGCL and XSimGCL show that noise added to embeddings improves contrastive recommendation mainly by spreading representations uniformly. XSimGCL also reports mixed results on non-graph backbones such as matrix factorization and plain feed-forward networks, so the gain may not carry over to transformer classifiers. The tension we target is specific. DHRD and Abstract-CoT both find that disrupting the order of reasoning destroys its benefit, which suggests isotropic noise should not substitute for structured reasoning. The matched-norm isotropic ablation tests this directly.

Why not just let the student write its own rationales for the unannotated examples? Self-generated pseudo-rationales cost student decoding tokens and training tokens on every example, so they reintroduce the overhead we are removing. There is also evidence that weak sources need guarding. SCOTT needs contrastive decoding and counterfactual training to stop students ignoring their rationales. Masked Distillation shows that fully internalized reasoning collapses on Countdown, to 41.7% versus a teacher's 87.3%. We include student self-rationalization as a baseline at matched GPU-hours. If it wins, latent imputation is the wrong lever.

Proposed Experiments

Tasks and setup. We use the classification tasks where both inference-free rationale distillation and our cost argument apply. These are the SuperGLUE tasks BoolQ, COPA, CB, and MultiRC from DHRD, together with e-SNLI, ANLI, and CommonsenseQA from Distilling Step-by-Step. StrategyQA (Did Aristotle Use a Laptop?) serves as a multi-hop stress test; its paper reports about 66% for a multi-task RoBERTa baseline and 87% for humans. Seed fractions are 5%, 12.5%, 25%, and 100%. One open-weight teacher writes all rationales, conditioned on the gold label to limit unfaithful text, following the practice SCOTT describes. Metrics are accuracy, teacher tokens, training tokens, and GPU-hours, with five seeds per configuration.

Decisive checks. Three outcomes decide the claim.

The first is the gap-recovery ratio: how much of the accuracy gap between seed-only DHRD and full DHRD the method closes. A ratio above one half at 12.5% seed would support the core claim.

The second is the pair of ablations. Directed imputation must beat both isotropic noise and shuffled-seed imputation. If it does not, the result is a regularization finding, which is useful but different.

The third is a robustness check on ANLI and the hard NLI sets used by DISCO. There we test whether imputed reasoning transfers like real rationales do, or only fits in-distribution labels.

A secondary analysis measures how much of the variance in $h_{qr}-h_q$ the top-$k$ directions explain on each task. Strong gains should coincide with low-rank, consistent shifts, and failures with diffuse ones. This would give a cheap pre-check for when a practitioner can skip most of the teacher bill.