A common recipe for building a small, fast text classifier when almost no labeled data exists is to have a large language model (the "teacher") write training examples aimed at the small model's (the "student's") current mistakes, retrain the student, and repeat. Systems like PGKD and EvoKD show this closed loop beats generating one fixed synthetic dataset. Nobody has asked the accounting question behind it, though. At a fixed teacher budget, is it better to pay the teacher to write new text or to label real unlabeled text that already sits near the student's mistakes? A second problem sits inside the loop. The examples the student finds hardest are also the ones the teacher is most likely to label wrongly, and today's loops need a labeled validation set to find those mistakes, which is exactly what label scarcity takes away.
This note proposes a closed-loop distillation method in which every teacher token is charged against one budget. The student's weak regions are found on unlabeled real data without any validation labels. Real unlabeled text is used first, and the teacher writes new text only where the real data is too thin. Points are sent to the teacher with a sampling rule that deliberately avoids the regions where the teacher's own labels are unreliable. The deliverable is a method and a measurement: accuracy-versus-teacher-token curves that show, for the first time, where closed-loop synthesis actually pays for itself.
Core Claim
Under label scarcity (one to five gold labels per class), most of the value of closed-loop generative distillation comes from choosing which inputs the teacher labels, not from the teacher inventing inputs. If that holds, a loop that draws inputs from the real unlabeled pool, writes text only where the pool lacks coverage, and discounts regions where the teacher is unreliable will match or beat generation-heavy loops like PGKD at a fraction of the teacher tokens. It will also do so without a labeled validation set. The claim is falsifiable: if generation-heavy loops dominate at every matched budget, the answer to the accounting question is the reverse, and that is worth knowing too.
The motivating evidence comes from four sources. Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale prompts a teacher with a validation-based error report and mined hard negatives to write targeted examples, but it never reports cumulative teacher cost against a static corpus. Evolving Knowledge Distillation with Large Language Models and Active Learning runs a similar loop from one shot, uses separate teacher calls for writing and labeling, and finds its advantage shrinks as data grows. That shrinkage hints that the writing budget is being spent inefficiently. Robust Active Distillation supplies the key constraint: on a fixed pool, the points where the student is least sure are the points the teacher most often mislabels. It gives a fast sampling rule that shifts mass toward "moderately hard" points, but that rule needs a validation estimate of teacher error and never generates anything. Active Mixup is the closest precedent in spirit. It builds many cheap candidates, lets the student pick, and pays the black-box teacher only for labels. Its candidates are pixel interpolations, though, which do not transfer to text.
Research Program
The loop runs in rounds. It starts from $n$ gold labels per class and a large unlabeled pool from the target domain. Everything is embedded with a fixed sentence encoder.
The first step locates student weaknesses without validation labels. Each unlabeled point gets a weakness score from two free signals. The first is the student's margin, meaning how close its top two class probabilities are. The second is disagreement between the student and a nearest-neighbor vote over points the teacher has already labeled. Points are then grouped into regions by clustering in embedding space, and each region's weakness is its mean score. This replaces PGKD's validation report.
The second step estimates teacher reliability without validation labels. The Robust Active Distillation rule needs to know how teacher error varies with student margin. We estimate this from teacher self-consistency. On a small, budget-charged calibration sample from each margin band, the teacher labels each point several times, under sampled decoding and two paraphrased prompts. The disagreement rate serves as the error proxy. Whether this proxy tracks true teacher error is an explicit experiment, described below, because the method depends on it.
The third step decides where to spend teacher tokens. Let $w(r)$ be region $r$'s weakness and $\hat{e}(r)$ its estimated teacher error. Points are sampled for teacher labeling with probability
$$p(x) \propto w(r_x)\,\big(1 - \hat{e}(r_x)\big)^{\gamma},$$
where $r_x$ is $x$'s region and $\gamma \ge 0$ controls how strongly unreliable regions are discounted. Setting $\gamma = 0$ recovers pure hardness-seeking, the behavior of PGKD, EvoKD and Lion. When a high-weight region contains fewer real unlabeled points than its allocation, the teacher is prompted to write new examples. The prompt is conditioned on that region's nearest real points and cluster description. The teacher writes these examples and labels them in a single call rather than EvoKD's two, and single versus separate calls is ablated.
The fourth step is the accounting. Every call is charged to one budget,
$$B = T_{\text{label}} + T_{\text{write}} + T_{\text{calib}},$$
where the three terms are total teacher tokens spent on labeling, on writing new text, and on reliability calibration. Every method in the study is compared at identical $B$. This is also how we answer the accounting question directly: we sweep a cap on $T_{\text{write}}/B$ from 0 (pool only) to 1 (generation only).
Baselines
The naive reference points are two. The first is a student fine-tuned only on the $n$ gold labels per class. The second is a student trained on teacher labels for a uniformly random sample of the unlabeled pool, at budget $B$. The second is the honest "do nothing clever" point, since random labeling of real data is often surprisingly strong.
The strongest existing closed-loop method is PGKD. It runs with the same student (BERT-base, as in the paper), the same teacher model, the same starting gold labels and the same token budget. Because PGKD needs a validation report, it gets its validation split carved out of the gold labels, which is what a practitioner would have to do. We also run an oracle variant with a large gold validation set, which bounds what validation-free diagnosis gives up. The reported reference number is that PGKD raised BERT-base accuracy on the 335-class AMZN Reviews dataset "from approximately 32% to over 44%" (PGKD). Its label regime and token spend differ from ours, so we reproduce it rather than quote it across settings. Beating PGKD means higher accuracy at equal $B$, or equal accuracy at clearly lower $B$ (we target at most half), with non-overlapping confidence intervals across at least five seed draws.
The EvoKD-style loop, with hard and easy mixing and separate writing and labeling calls, runs under the same budget. The reference is EvoKD's report of "an F1 score improvement of over 20% compared to AugGPT" on Amazon sentiment and "approximately 90% of the performance of models trained on complete datasets" at one shot (EvoKD).
Robust Active Distillation applied to the unlabeled pool is the pool-only competitor. It runs with its original validation-based error estimate from the gold labels, and there is no generation. No reported result was found on Lacuna for RAD on text classification. A static baseline has the teacher write one fixed synthetic corpus of cost $B$ upfront, as in AugGPT-style augmentation, and label it.
Three ablations isolate the proposal's ingredients. Setting $\gamma = 0$ tests whether reliability hedging matters in a generative loop. Setting $T_{\text{write}} = 0$ tests whether writing ever pays. Swapping our weakness score for a gold-validation report tests whether validation-free diagnosis costs accuracy.
Why not just…?
Why not just generate one large static synthetic dataset with the same budget? This is the cheapest thing to engineer, and it is the baseline most likely to be underrated. DISCO found 75,000 filtered counterfactuals beat over a million unguided ones. Beyond Quantity documents early plateaus from redundancy in volume-scaled synthetic data. Both suggest untargeted volume wastes budget. Neither compared against a loop at matched cost, however, so the static arm at identical $B$ settles it directly.
Why not just have the teacher label actively chosen real unlabeled text and never generate? This is the real threat, and the proposal is built around it. If the $T_{\text{write}} = 0$ setting matches the full method at every budget, the conclusion is that generative loops like PGKD mostly buy their gains through input selection. The writing machinery would then be unnecessary wherever an unlabeled pool exists. We report that outcome as a primary result, not a failure. We expect writing to help only on high-cardinality tasks where rare classes are thin in the pool (AMZN 335, HuffPost 41), and the per-class breakdown will show whether that is where it helps.
Why not just use the language model directly with a few well-chosen in-context examples? Selective Annotation Makes Language Models Better Few-Shot Learners shows its vote-k selection gives "a 12.9% relative gain over random selection" with 18 examples, so strong in-context performance is plausible. The deployment gap is large, though: PGKD reports its distilled student is "130x faster than LLMs on CPU" and "25x less expensive" at inference. We report teacher in-context accuracy with vote-k demonstrations as the ceiling the student is trying to approach, not as a competitor for the deployment setting.
Why not just self-train the student on its own pseudo-labels? This costs no teacher tokens at all. It is known to reinforce the model's own errors, and RAD's analysis shows the boundary is where errors concentrate. We include it as the $B = 0$ point on every curve, so the value of each teacher token can be read off directly.
Proposed Experiments
The benchmarks are the PGKD suite (AMZN Reviews 335 classes, Huffington Post 41, Yahoo Answers 10, AG-news 4) and EvoKD's Amazon sentiment and IMDB. All use one, two and five gold labels per class and budgets spanning roughly one to two orders of magnitude of teacher tokens. Gold labels for the full training sets are held out and used only for analysis, never by any method. The metric is test accuracy (macro-F1 for the sentiment sets) plotted against $B$, with the area under that curve as the summary number.
Three checks decide the proposal. The first tests the reliability proxy. Using the held-out gold labels, we measure whether teacher self-consistency predicts actual teacher error across student-margin bands. We also test whether teacher error peaks near the student's boundary for generated examples, as RAD found for pool examples. If the proxy's rank correlation with true error is weak, the hedging step is unsupported, and we fall back to reporting the $\gamma$ ablation as a negative result. The second is the write-fraction sweep: the budget share given to writing that maximizes accuracy, per dataset and per class-frequency bucket, which answers the accounting question directly. The third tests confirmation bias. We compare teacher label error on the points actually used for training under $\gamma = 0$ versus the chosen $\gamma$, and check whether lower label error translates into student accuracy. That link is the step the whole argument rests on.
References
- Performance-Guided LLM Knowledge Distillation for Efficient Text Classification at Scale
- Evolving Knowledge Distillation with Large Language Models and Active Learning
- Robust Active Distillation
- Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation From a Blackbox Model
- DISCO: Distilling Counterfactuals with Large Language Models
- Beyond Quantity: Trajectory Diversity Scaling for Code Agents
- Selective Annotation Makes Language Models Better Few-Shot Learners