Recycling Filter Rejects as a Continuation Schedule for Synthetic-Data Students

Pipelines that train small models on data written by large models throw much of that data away. A large "teacher" model writes candidate training examples, a verifier checks each one, and anything that fails is deleted. The teacher forward passes spent on rejected candidates are pure overhead. Separately, work on hard, bumpy training objectives shows that small models train better when the target starts blurry and noisy and is sharpened over time. This technique is called continuation: optimize a smoothed version of the problem first, then gradually restore the original.

This proposal joins the two. We treat the verifier's rejects as the smoothing signal. Early in training, rejected candidates enter the loss with soft targets and a weight that shrinks on a fixed schedule until only verified data remains. The claim is concrete and falsifiable. At a fixed budget of teacher calls, a student trained with annealed rejects should match or beat a student trained on verified data alone. It should also beat a student that gets its smoothing from artificial target softening. If the second comparison fails, the rejects carry no information beyond generic noise, and the idea is wrong.

Core Claim

Two strands of prior work motivate the proposal. Policy Gradient Algorithms Implicitly Optimize by Continuation proves that injecting variance into a policy convolves a multi-modal return surface into a smoother one, removing local traps. Continuation KD carries the same logic into distillation: it softens teacher outputs and widens a hinge margin early, then sharpens both. It reports 90.21% versus 88.45% for vanilla distillation on CIFAR-10 and raises CoLA from 60.97 to 63.75. In both works the smoothing is manufactured, either from Gaussian noise on actions or from temperature on clean-input targets.

The synthetic-data pipelines that supply the application regime go the opposite way and remove noise.

  • DISCO has GPT-3 write counterfactual NLI edits and keeps only those a DeBERTa-v2 teacher confidently flips.
  • Promptagator has FLAN write retrieval queries and keeps only those passing a round-trip retrieval check.

Both show that unfiltered data hurts. DISCO's 75,000 filtered examples beat prior baselines using over a million noisier ones. The research file characterizes such cascades as discarding up to 50–70% of generations, but that figure is the compiler's summary rather than a verified per-paper number, so we will measure rejection rates ourselves.

Our hypothesis is that rejects are not uniformly bad. Many are near-boundary or label-ambiguous inputs, for example an edit that only half-flipped the label, or a query that retrieves a neighbor of the source document. Near-boundary inputs are what a data-manifold version of Gaussian smoothing would sample. Continuation KD blurs targets on fixed clean inputs. We instead blur the input distribution with real, already-paid-for teacher outputs, and then anneal that blur away. That difference is what would be new, and it is also exactly what the key ablation tests.

Research Program

Scoring and targets. Every generated candidate $x$ receives a verifier score $s(x) \in [0,1]$. In DISCO this is the teacher's probability of the intended flipped label; in Promptagator it is a normalized round-trip rank of the source document. Candidates above the standard threshold are "kept" and trained with their intended label. Rejected candidates get a soft target $q_\tau(x)$: the verifier's full label distribution, or for retrieval a softmax over the round-trip retrieval scores, flattened by temperature $\tau$.

Loss and schedule. The student minimizes

$$\mathcal{L}(t) = \mathbb{E}_{x \in \text{kept}}\,\ell(f(x), y) + w(t)\,\mathbb{E}_{x \in \text{rejected}}\; s(x)\,\mathrm{KL}\big(q_{\tau(t)}(x)\,\|\,f(x)\big),$$

where $f$ is the student, $\ell$ is the usual task loss, and $t$ is the training step. The weight $w(t)$ decays from $w_0$ to zero over the first fraction of training, while $\tau(t)$ decays toward 1. The objective therefore starts as a blurred problem over a wider input set and ends as exactly the standard filtered objective. This gives continuation's guarantee in spirit: we never change the final target.

Diagnostics. Beyond the loss, we will measure whether rejects actually smooth the student's landscape. We will record loss along random and top-Hessian directions at matched checkpoints, sharpness at convergence, and seed-to-seed variance of final accuracy. Seed variance is a practical proxy for basin trapping.

Compute accounting. The overhead side is handled by reporting every result against teacher calls consumed, not examples kept. That way "recycling" is credited only for compute that was actually saved.

Baselines

The naive reference point is training on all generations with their intended labels and no filter. DISCO already shows this is worse than filtering, so it bounds from below.

The standard method is the strict-filter recipe of each source paper, rerun by us with open teachers of comparable role. text-davinci-002 and FLAN 137B are not reproducible, so absolute numbers will differ from the papers. The published anchors are these. DISCO reports a 6% absolute gain on NLI stress tests and over 20% and 30% improvements on SNLI-hard and WANLI. Promptagator reports 47.8 nDCG@10 averaged over BEIR tasks with 8 examples, versus 45.5 zero-shot, and a 1.2-point average margin over ColBERT v2 and SPLADE v2. We will first reproduce the direction of these effects before claiming anything.

The strongest existing competitor is Continuation KD applied to the kept set only, with its annealed margin and softened teacher targets. It delivers continuation without rejects. It is run on the same kept data, with the same student and step count, and the same hyperparameter search budget (identical number of trials over its margin and decay versus our $w_0$ and decay length).

The key ablation replaces each rejected input with a matched-count, matched-weight synthetic perturbation of a kept input:

  • for NLI, token-dropped copies with uniform soft targets;
  • for retrieval, random-negative soft labels.

This keeps the schedule and removes the rejects' content. A second ablation keeps rejects with verifier soft labels at a constant weight and no annealing, isolating the annealing itself. Beating a baseline means a gain on the out-of-distribution metrics (HANS, stress tests, WANLI, SNLI-hard; BEIR nDCG@10) that exceeds seed standard deviation across five seeds at equal teacher calls.

Why not just…?

Why not just generate more data and keep filtering? This spends the overhead we are trying to cut, and the file's evidence says volume saturates. TDScaling finds an early bottleneck from repetitive synthetic paths, and DISCO's 75k filtered examples beat a million noisy ones. We test this directly by plotting student accuracy against teacher calls for strict filtering at increasing budgets. Our method must sit above that curve, not merely at one point on it.

Why not just lower the filter threshold? A looser threshold admits more rejects but with hard intended labels and no schedule. That is the naive baseline pushed partway, and we sweep it. If a tuned threshold matches us, then the annealed soft target adds nothing and we will report that.

Why not just relabel rejects with the teacher's prediction and train on them as ordinary data? This is a real threat. A DISCO edit that failed to flip is often a valid example of the original label. The constant-weight soft-label ablation is exactly this alternative. The proposal's contribution survives only if annealing beats it on out-of-distribution sets, the regime where continuation arguments predict an effect. If relabeling wins, the honest conclusion is that rejects are useful data but not a continuation signal. That would still be a practical overhead result.

Why not just use Continuation KD on the clean data? If blurred targets alone give all the benefit, the rejects are redundant. The synthetic-perturbation ablation and the Continuation KD baseline together settle this.

Proposed Experiments and Decisive Checks

We run two testbeds.

  • NLI. Following DISCO, train small students (a DeBERTa-small class model, where capacity gaps should matter most) on SNLI-seeded counterfactuals. Evaluate on MNLI-hard, QNLI, the NLI Stress Test, HANS, WANLI, and SNLI-hard.
  • Retrieval. Following Promptagator, train a small dual encoder per BEIR task (ArguAna, SciFact, NFCorpus, FiQA-sized corpora first) from 8 examples. Evaluate by nDCG@10.

Each configuration uses five seeds and the same teacher-call budgets.

The decisive check is a two-part ordering.

  1. At equal teacher calls, annealed rejects must beat the strict filter.
  2. Annealed rejects must also beat the synthetic-perturbation ablation.

If only the first holds, rejects help as data, not as smoothing. If neither holds, the hypothesis is refuted.

A secondary check is mechanistic. The gains should be largest where the landscape diagnostics show the strict-filter student is sharpest or most seed-variable, and should shrink as student size grows. That scaling pattern would mirror the overparameterization result in Liu et al., where wider networks turn traps into saddles.

We report no numbers for our method in advance. No reported result exists on Lacuna for training on rejected synthetic candidates in either setting.