Spending the Generation Budget Where the Student Fails: Lasso-Targeted Synthetic Data

Training a small model on data written by a large one costs a lot of money, and most of that money is wasted. Pipelines typically generate hundreds of thousands of examples, then discard most of them with filters, judges or coverage algorithms. The rejected samples have already been paid for. This proposal moves selection to before generation. We look inside a model through a sparse autoencoder, a small network that re-expresses a model's internal activations as a large dictionary of mostly-inactive, individually interpretable "features". On a small real diagnosis set, we fit a sparse linear regression (the lasso, which forces most coefficients to exactly zero) that predicts which examples the current student model gets wrong from which features those examples activate. The handful of features that survive are the student's blind spots. We spend the generation budget only on those, in proportion to how much each one predicts error, and we refit after every round.

In plain terms, the student's mistakes tell us which few internal concepts it lacks, and a convex sparse fit names them before we pay a teacher to write anything. The claim is that this reaches the downstream quality of the strongest feature-targeted synthesis method at a markedly smaller generation-token budget. It should also beat post-hoc pruning at equal budget, because it never generates the redundant samples that pruning later removes.

Where this sits

The closest work is Less is Enough. It measures diversity as the fraction of task-relevant sparse-autoencoder features a dataset activates, a quantity it calls Feature Activation Coverage (FAC). It shows FAC correlates with downstream performance at r = 0.95. It then generates data for features present in an anchor corpus but missing from the seed set, matching MAGPIE on AlpacaEval 2.0 with 2,000 samples instead of 300,000. This paper motivates us, because it shows feature space is the right place to target generation. It is also our main competitor.

Its selection rule is a set difference, and that rule has two gaps. It treats every missing feature as equally worth covering. It also ignores whether the student already handles a feature's content through other features. Our rule is supervised by the student's own errors and weights features by effect size. It is also iterative, so a feature is retired once the student stops failing on it. We also drop FAC's dependence on a GPT-4o-mini judge to name features, and instead prompt with real contrastive exemplars.

We take the method from Sparse Relaxed-Lasso Steering. That paper selects about 300 of about 7,800 autoencoder features for diffusion-model editing by solving a lasso and then refitting ridge on the chosen support to undo shrinkage. That work operates at inference time and finds one steering direction. We reuse the same two-stage estimator for a different job: allocating a synthesis budget across many features for training a student.

Less is More: Adaptive Coverage constrains the design and supplies the pruning baseline. Its Adaptive Coverage Sampling keeps 10–30% of a synthetic corpus without loss. That result shows most generated volume is redundant, but the method must generate, embed and graph the full corpus first. The post-hoc filters in DISCO, BARRED and Promptagator share that sunk-cost structure. Promptagator's note that round-trip filtering was too strict on small NFCorpus warns that aggressive rejection can remove informative boundary cases. Our error-driven targeting is meant to seek out exactly those cases rather than filter them away.

Core Claim

For a student and task, let $a_i \in \mathbb{R}^p$ be the pooled autoencoder activations of diagnosis example $i$. Here $p$ is in the tens of thousands, while the diagnosis set has only a few thousand examples. Let $\ell_i$ be the student's loss on that example. We solve

$$\hat\beta = \arg\min_{\beta \ge 0,\, b}\ \frac{1}{2n}\sum_{i=1}^{n}\left(\ell_i - b - a_i^\top \beta\right)^2 + \lambda \lVert \beta \rVert_1,$$

where $n$ is the number of diagnosis examples, $b$ is an intercept, and $\lambda$ controls sparsity. The nonnegativity constraint keeps only features whose presence raises loss. We then refit ridge on the support $S = \{j : \hat\beta_j > 0\}$ to obtain debiased coefficients $\tilde\beta_j$. Each feature $j \in S$ receives a share $B_j \propto \tilde\beta_j$ of the round's generation budget.

This is the regime the lasso was built for, with many more candidate predictors than observations. Autoencoder features co-fire heavily, so the lasso's known habit of choosing one feature arbitrarily from a correlated group is a real risk. We address it with bootstrap stability selection: a feature is kept only if it is selected in most resamples.

Research Program

Each round follows the same cycle. First, run the frozen autoencoder model over a held-out split of the real training data and record student losses. Second, fit the stability-selected relaxed lasso. Third, generate for each surviving feature. The generator receives the $k$ real snippets that activate the feature most strongly and $k$ lexically similar snippets that do not, together with their true labels. It is asked for new examples that share the contrast.

Fourth, before paying for training, check each new sample in one cheap forward pass to confirm that it actually fires its target feature. This is the only filter, and it costs far less than a judge or a debate. Fifth, retrain the student and repeat until the refit support is empty or the budget runs out.

We will analyze three things. The first is how quickly the support shrinks across rounds, which tells us whether the method self-terminates. The second is whether $\tilde\beta$ predicts per-feature marginal gain on held-out data. The third is cost, tracked as teacher tokens plus every overhead the method adds: autoencoder passes, the lasso solves and the verification passes.

Baselines

The naive reference is standard generation at matched teacher-token budget, with seed-conditioned prompting and uniform random subsampling. It shows what budget alone buys.

The strongest existing method is FAC Synthesis from Less is Enough. It reports an AUPRC improvement of +23.63% over its baseline on ToxicChat, nearly four points above the next synthesis method, and 76.22% overall accuracy on RewardBench, a +13.32% gain. We will rerun it with the same autoencoder, generator, student and total token budget. We will replace its GPT-4o-mini labeling with the same exemplar prompting, so that only the selection rule differs. We will also report it with its judge intact. Beating it means matching its ToxicChat AUPRC with at most half its generation tokens, or exceeding it at equal tokens, across three seeds.

The generate-then-prune competitor is Adaptive Coverage Sampling Less is More. It is given a full corpus and allowed to keep its best fraction, with all of its generation and embedding charged to its cost. On SST2 it is reported to match the full corpus with 10% of the data. Our method wins if it reaches ACS's retained-subset quality while generating only roughly the retained volume.

Three ablations isolate the key ingredient. The first swaps the error-regression lasso for FAC's set difference while keeping everything else, which tests the selection signal. The second swaps it for univariate top-$k$ correlation with loss, which tests the value of joint sparse fitting under collinearity. The third drops the ridge refit and allocates by raw $\hat\beta$, which tests debiasing. SRLS's result that relaxed-lasso selection beats heuristic top-$k$/PCA (harmonic mean 3.97 vs 3.89, with 25x fewer features) gives reason to expect the second and third ablations to lose, but that result was in image editing and has to be shown again here.

Why not just…?

Why not just generate more and randomly subsample? Adaptive Coverage reports that ACS consistently beats random sampling on SST2, FewRel and CrossNER. Beyond Quantity finds an early bottleneck where volume scaling stalls while diversity scaling keeps improving. Our matched-budget naive baseline checks this again on ToxicChat.

Why not just use FAC's missing-feature rule, which already works? This is the real threat. If missing-from-seed features coincide with error-predictive features, our method is a more elaborate route to the same set. We will measure the overlap of the two supports directly. The first ablation settles the question at equal budget. If overlap is high and gains vanish, the contribution shrinks to judge-free prompting and adaptive stopping, and we will report it as such.

Why not just cluster the student's failing examples in embedding space and prompt with them? This hard-example-mining approach needs no autoencoder, and it may work. Less is Enough argues that surface representations are weaker signals because lexically diverse text can fire the same internal features. That argument, however, concerns diversity metrics, not error targeting. We therefore run this baseline at matched budget. If it ties, the autoencoder is unnecessary and we will say so.

Why not just filter with self-refinement instead of feature verification? BARRED reports that single-model critique performed worse than no verification. Our verification is a feature-activation check and not a critique, and we keep it only if removing it hurts.

Proposed Experiments

The primary testbed is ToxicChat, which has about 5k real training queries. From these we hold out a diagnosis split. The student is a RoBERTa-scale classifier and the metrics are AUPRC and F1. The file reports F1 72.1 for RoBERTa trained on the real training set, which anchors the student's scale. The secondary testbed is RewardBench, measured by accuracy, where FAC reports 76.22%. For the comparison with pruning we use SST2 and CrossNER against ACS.

The decisive plot is downstream metric against cumulative teacher tokens, including all overhead, for every method. The proposal succeeds if its curve dominates FAC's and ACS's across the budget range. A second decisive check asks whether ridge-refit coefficients rank the marginal value of held-out features, measured as Spearman correlation between $\tilde\beta_j$ and the gain from generating for feature $j$ alone. If they do not, the budget-allocation part of the claim fails even if selection works.

We will also test stability. If bootstrap stability selection yields supports that change substantially across diagnosis resamples, targeting is noise-driven. In that case we would move to group-level selection over clusters of co-firing features.