Does the Student's Shape Decide Whether Synthetic Data Needs Filtering?

A common way to build a small, fast model for a specialized task is to have a very large language model write the training data. A huge "teacher" model invents search queries, labels, or counterfactual examples, and a compact "student" model learns from them. A large part of the cost lies in cleaning that data. Raw generations contain hallucinated or mismatched examples, so pipelines add verification stages. One example is training a temporary model and discarding queries that fail to retrieve their source document. Another is running every generation through a separate judge model. Each stage multiplies the compute spent per usable example.

This note proposes that part of that cleaning cost is a property of the student, not the data. Two networks with the same parameter count can differ in shape: a few wide layers, or many narrow ones. Theory and small-scale evidence suggest that deep, narrow networks fit noise more readily when the effective sample is limited, while wide, shallow ones generalize more conservatively. If this holds for distilled students, then a wide and shallow student trained on unfiltered synthetic data could match a standard-shaped student trained on filtered data. Because shallow models are also much faster at inference, the same choice would save money twice: once by skipping the filter, and again at serving time. We will measure, at fixed parameter count, how much filtering helps as a function of the student's depth-to-width ratio. We will also check whether ordinary regularization already gives the same benefit.

Core Claim

The central quantity is the filter gain of a student shape. For a student with aspect ratio $R$ (hidden width divided by number of layers) and a fixed parameter count, define

$$\Delta(R) = S_{\text{filtered}}(R) - S_{\text{raw}}(R),$$

where $S$ is downstream quality (nDCG@10 for retrieval). $S_{\text{filtered}}$ is quality after training on the verified synthetic corpus, and $S_{\text{raw}}$ is quality after training on the unverified one generated from the same prompts. The claim is that $\Delta(R)$ shrinks as $R$ grows, meaning filtering matters less for wider, shallower students. The claim also has a practical half. At the filter-free operating point, a shallow student should reach the quality of the standard filtered pipeline while costing less end to end. Here "end to end" means

$$C_{\text{total}} = C_{\text{gen}} + C_{\text{filter}} + C_{\text{train}}(R) + T \cdot c_{\text{infer}}(R),$$

with $C_{\text{gen}}$ the teacher's generation cost, $C_{\text{filter}}$ the verification cost, $C_{\text{train}}$ the cost of training the student, and $T \cdot c_{\text{infer}}$ the per-query serving cost $c_{\text{infer}}$ over $T$ queries. This is a narrower and more falsifiable question than the general "co-optimize shape and synthetic data volume" gap. It isolates one cost term, filtering, that practitioners pay today without asking whether the student's shape created the need for it.

Why This Is Plausible, and What Pushes Against It

Three results motivate the hypothesis. Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss splits generalization error into approximation and sampling terms, with the sampling term scaling like $W^2/M \log M$ for $W$ parameters and $M$ samples. In that paper a depth-1, width-20 network beats a depth-4, width-10 network at 1,000 collocation points, and the order reverses at 10,000. Its "data" are noise-free PDE collocation points, which the paper treats as a free oracle. We ask the analogous question when $M$ is large but a fraction of the samples are wrong. “No Free Lunch” in Neural Architectures? proves, for networks viewed as computation graphs, that deep and narrow topologies minimize training loss while wide and shallow ones generalize better and converge faster. This is exactly the split we would expect between memorizing bad labels and ignoring them. The solvable binary-neuron model of Optimal Architectures in a Solvable Model of Deep Networks adds a second side: depth there removes input noise layer by layer. That concerns corrupted inputs, not corrupted labels, but it warns that depth could help rather than hurt. Likewise, Yaras et al. argue that depth biases learning toward low-rank solutions, which could also act as noise robustness. The experiment therefore genuinely has two possible outcomes, and we will report the direction we find.

A further result constrains the design. Theory of Scaling Laws for In-Context Regression shows that depth becomes necessary when the task changes from prompt to prompt, because the network must run an iterative algorithm across its layers. Retrieval over heterogeneous queries may need depth for the same reason. If so, a too-shallow student will hit a capability wall that has nothing to do with noise. This is why we sweep shape continuously rather than comparing two extremes.

On the synthetic-data side, the filtering stages we want to remove come from Promptagator, which prompts a large model with 8 examples to write millions of queries and then applies a round-trip retrieval filter. DISCO reports that 75,000 judge-filtered counterfactuals beat more than a million unfiltered automated ones. Both papers use a fixed off-the-shelf student, a 110M dual encoder and RoBERTa-large respectively, so neither can tell whether the filter's value depends on that student. UDAPDR shows a different way to avoid filtering: it distills an ensemble of cross-encoder rerankers into ColBERTv2. Its noise tolerance comes from extra teacher models, not from the student's shape, so it is a competitor rather than a precursor. Finally, the inference half of the claim rests on Scaling Inference-Efficient Language Models and Decoding Speculative Decoding. The first finds up to a 3.5x latency gap at equal parameter count driven by depth. The second finds that a 5-layer wide pruned draft model gives 111% higher speculative throughput than Sheared-LLaMA-1.3B.

Research Program

We first build a controlled family of encoder students at about 110M parameters, matching Promptagator's student size. The family spans roughly six aspect ratios, from about 4 wide layers to about 24 narrow ones. All shapes are pretrained on the same corpus with the same token budget. This step is necessary because off-the-shelf checkpoints differ in more than shape. Depth must also not be confounded with poor tuning. Don't be lazy: CompleteP shows that standard residual scaling pushes deep transformers into a near-linear "lazy" regime and that μP does not transfer learning rates across depth. Its $L^{-1}$ residual scaling fixes both problems. We therefore pretrain every shape under CompleteP and transfer hyperparameters from a shallow proxy, and we keep a standard-parameterization copy of two shapes as an ablation.

Next, an open instruction-tuned teacher generates synthetic queries for the BEIR corpora used by Promptagator, using the same 8-example prompting. Each corpus exists in three versions: raw generations, round-trip filtered generations as in Promptagator, and raw generations with a controlled fraction (0%, 10%, 20%, 40%) of query–document pairs deliberately mismatched. The controlled version separates noise robustness from the side effect that filtering also shrinks and reshapes the dataset. That side effect is real: Promptagator reports over-filtering on small corpora such as NFCorpus. Every student shape is fine-tuned on every version. We record $\Delta(R)$, the slope of quality against injected noise for each shape, and the full $C_{\text{total}}$ including measured inference latency.

A second, smaller testbed checks generality outside retrieval. Following TrueTeacher, which labels 1.4M summaries with FLAN-PaLM 540B and distills them into T5-11B, we have an open teacher label summaries for factual consistency. Shape-swept classifiers are trained on raw labels and on labels filtered by teacher confidence, then evaluated on the TRUE benchmark by ROC-AUC. TRUE's inputs degrade beyond 200 tokens, so we report results stratified by length.

Baselines

The do-nothing reference is BM25 on the same BEIR datasets. It needs no generation, filtering, or training. BEIR describes it as a surprisingly strong zero-shot baseline that beats DPR, though the file gives no per-dataset numbers. We run it ourselves with the BEIR toolkit. Any synthetic pipeline that does not clearly beat BM25 on average is not worth its overhead.

The strongest existing method on this question is Promptagator's recipe: a standard BERT-base-shaped 110M dual encoder trained on round-trip filtered synthetic queries. Promptagator reports 47.8 nDCG@10 averaged over 11 BEIR datasets with 8 examples and FLAN 137B. Our open teacher will not reproduce that exact figure, so we reimplement the recipe with our teacher and treat our own reproduction as the bar. The comparison is fair because it uses the same generated corpus before filtering, the same pretraining tokens, and the same fine-tuning trial budget for every arm. Beating it means a wider, shallower student trained on raw data that comes within 0.5 nDCG@10 of this filtered baseline, or exceeds it, across at least three random seeds, while showing lower measured query latency and no filtering cost.

The filter-free competitor is UDAPDR-style multi-teacher distillation into ColBERTv2. UDAPDR reports raising LoTTE Lifestyle Success@5 from 64.5 to 74.8. We run it on LoTTE Lifestyle with our generated queries and count the cost of training its reranker ensemble in $C_{\text{total}}$. This answers whether shape is a cheaper route to noise tolerance than adding extra teacher models.

Two ablations isolate the key ingredient. The first trains the standard-shaped student on raw data with regularization tuned for noise: early stopping on a small held-out set of real queries, dropout, and label smoothing over in-batch negatives. It gets the same tuning budget as the shape sweep. The second retrains two shapes under standard parameterization instead of CompleteP, to show whether any depth penalty reflects lazy training rather than depth itself.

Why not just…?

Why not just regularize the standard student instead of changing its shape? This is the most serious threat. Early stopping and dropout are known to limit memorization of bad labels, and they cost almost nothing. If the tuned-regularization ablation closes the gap between raw and filtered training for the standard shape as well as the shallow shapes do, then shape is not the lever and the quality half of our claim fails. The latency argument from Scaling Inference-Efficient Language Models would survive on its own, but it would no longer depend on synthetic data. We designed the experiment so that this outcome is clearly visible and would be reported as the main result.

Why not just use multi-teacher distillation, which already removes filtering? UDAPDR does remove explicit filtering. It does so by training several DeBERTa-v3-Large rerankers on synthetic data, and that training is itself large overhead. Our cost accounting compares the two routes directly. If UDAPDR wins on quality at similar total cost, the two are complementary, and we will also test a shallow student as UDAPDR's distillation target.

Why not just make the filter cheaper, for example with a small round-trip retriever? A cheaper filter shrinks $C_{\text{filter}}$ but keeps its failure mode. Promptagator itself notes that over-aggressive filtering on small corpora like NFCorpus throws away needed domain vocabulary. The controlled-noise experiment shows whether a filter-free shallow student avoids that sample starvation. We will also report results with a cheap filter so readers can compare.

Why not just train a standard student and quantize or prune it afterward? Compression lowers memory and arithmetic cost but keeps the layer-by-layer sequential dependency that dominates latency. Decoding Speculative Decoding finds that latency tracks depth rather than parameter count. More importantly, compression after training does nothing about how the student absorbs noisy data, which is the question here.

Decisive Checks

The hypothesis predicts that $\Delta(R)$ falls monotonically with aspect ratio and that the noise-slope curves fan out, with deep students losing quality faster as injected mismatch grows. It is refuted if the curves are parallel, or if deep students degrade more slowly, as the noise-cleaning picture of Optimal Architectures in a Solvable Model of Deep Networks would suggest. A non-monotone result, with very shallow students losing on query-diverse datasets such as HotpotQA or ArguAna regardless of noise, would match the in-context-regression prediction that diverse tasks need depth. It would locate a best shape rather than an extreme. The practical claim succeeds only if some shape trained on raw data matches our reproduced filtered Promptagator baseline within 0.5 nDCG@10, stays clearly above BM25, and does so at lower $C_{\text{total}}$ under realistic query volumes $T$. The claim must also hold after the tuned-regularization ablation is given the same chance. The TRUE testbed then shows whether the pattern carries over from contrastive retrieval to classification with noisy labels.