Spending the Generation Budget Where the Sphere Is Empty

When a large language model writes training data for a smaller model, most of the cost is in the generation calls themselves. Many of those calls are wasted: the model produces near-duplicates of what it has already written, and later filters throw them away. This proposal decides before each call which seed it should be spent on. A seed is a document to write queries for, or an attribute combination to write examples from. The decision rule comes from hyperspherical energy minimization, a family of methods that treat normalized embedding vectors as charged particles on a sphere and push them apart so they cover it evenly. Each time, we pick the seed whose predicted output would most reduce the "energy" of the synthetic set built so far. We then correct that prediction with the embedding of what was actually generated.

Prior work in this area does one of two things. It selects from data that already exists, or it generates a large pool and prunes it afterwards. Neither reduces the generation bill itself. The bet here is that, in retrieval and text classification, where a generation will land in embedding space is predictable enough from its seed that we can skip most calls. The target is to match full-corpus synthetic training at a small fraction of the calls.

Core Claim

Under a fixed budget of generation calls, a greedy rule that allocates calls to seeds by minimizing an energy on the sphere will give a better student model than random allocation or farthest-point selection. The rule has two parts: matched points attract toward the target corpus, and selected synthetic points repel each other. A cheap surrogate predicts where each output will land, and that prediction is corrected online with the actual output embeddings.

The attraction term matters. High-dimensional Analysis of Synthetic Data Selection shows that a mismatch in covariance between synthetic and real data, not a shift in the mean, drives excess risk. Pure repulsion, which pushes points toward uniform coverage of the whole sphere, would deliberately create exactly that kind of covariance mismatch. We therefore use the energy-distance form. Let $S$ be the selected synthetic embeddings, $C$ a sample of target-corpus embeddings, and $k_s(x,y)=\|x-y\|^{-s}$ the Riesz kernel with exponent $s$, which sets how sharply nearby points repel. The objective is

$$ \mathcal{E}_s(S;C)=\frac{1}{|S|^2}\sum_{x,x'\in S}k_s(x,x')-\frac{2}{|S||C|}\sum_{x\in S,\,c\in C}k_s(x,c). $$

The first term repels selected points from each other, in the style of minimum hyperspherical energy. The second pulls them toward where the real corpus lives. This mirrors the split in Hyper-SET, which derives Transformer blocks from one mass-covering objective and one mode-seeking objective.

The exponent $s$ controls what the rule becomes. As $s\to\infty$, the repulsion term reduces to max-min packing, i.e. farthest-point or k-center selection, the regime used by the Tammes-style objective of MMA Regularization. Sweeping $s$ therefore directly tests whether a whole-configuration energy beats plain max-min selection.

Relation to Prior Work

Data-Efficient Learning via Minimizing Hyperspherical Energy motivates the work and also supplies a baseline. Its method, MHEAL, uses spherical k-means plus max-min selection to choose real, unlabeled points for human labeling. It reaches 54.09% on CIFAR-10 with 1,000 labels, against 32.45% for k-means selection. MHEAL never predicts or steers the output of a generator, and it has no term tying the selection to a target distribution. We supply both.

Learning towards Minimum Hyperspherical Energy and Regularizing Neural Networks via Minimizing Hyperspherical Energy supply the method. The first gives the energy itself. The second gives compressive random projections, which we use to keep energy evaluation cheap. Learning with Hyperspherical Uniformity supplies the virtual-antipode trick. We test that trick as an option, since embeddings of generated text occupy a narrow cone and may not fill the sphere.

Three lines of work attack generation overhead without geometry, and all of them constrain our design:

  • Covariance Matching prunes an already-generated pool in a 32-dimensional principal-component subspace. It reports 54.00% versus 49.38% for random selection on CIFAR-10, but it pays for the full pool first.
  • AttrPrompt reaches parity with simple class-conditional prompting at 5% of the query budget using hand-designed discrete attributes. It shows that steering before generation is where the large savings lie, but its attributes are discrete and require human seeding.
  • QDGS searches a generator's latent space evolutionarily. This moves the overhead into the search rather than removing it, and it is a cost we must avoid reintroducing.

TDScaling shows that steering by diversity beats scaling up quantity, but it measures diversity symbolically on tool graphs rather than geometrically.

Finally, Promptodile shows that a 3B–14B open model with no round-trip filtering already matches the original Promptagator. It sets our cost floor: we must save calls relative to an already-cheap pipeline.

Research Program

The system has three parts.

Surrogate. For each seed we predict where its output will land as a unit vector in a fixed encoder's embedding space. For retrieval, the seed is a document. The prediction is the document's embedding under unsupervised E5, shifted by a mean query-minus-document offset estimated from the few-shot examples. For classification, the seed is a label–attribute prompt. The prediction is the running mean embedding of earlier outputs from nearby prompts, falling back to the prompt's own embedding.

Greedy allocation. At each step we choose the seed whose predicted output most lowers $\mathcal{E}_s$, accounting for seeds already used. We then call the generator, embed the real output, and replace the prediction with it. Each step updates the energy incrementally against a random sample of the corpus and of the selected set, after a random projection in the style of the compressive method. This keeps the per-step cost linear rather than quadratic in the set size.

Stopping rule. A seed may receive repeat calls only while its marginal energy reduction stays above that of the best unused seed. This replaces a fixed "k queries per document" rule.

All wall-clock and floating-point costs we report include corpus embedding and selection overhead, so that geometry is not credited with savings it does not deliver. We also measure how well the surrogate predicts. The cosine between predicted and actual output embeddings is itself a result: if it is near the similarity between random pairs, the pre-generation premise fails.

Baselines

The naive reference point is full-corpus Promptodile. Llama-3.2-3B generates unfiltered queries for every document, reported at 45.1 average nDCG@10 on BEIR, alongside 46.2 for Phi-3-medium and 48.1 when the retriever is initialized from E5 Base (unsupervised) Study on LLMs for Promptagator-Style Dense Retriever Training. It needs no selection machinery, so any method must justify its overhead against it. The number of queries per document in that setup is not given in the file, so we will regenerate it ourselves to count calls exactly.

The second baseline is random allocation of the same reduced call budget, which tests whether selection matters at all at that budget.

The strongest geometric competitor is k-center greedy on the predicted embeddings, which corresponds to the $s\to\infty$ limit. We also run MHEAL as published, with spherical k-means and max-min selection over seeds. Both get the same surrogate, encoder, and budget, so the only difference is the selection rule.

The strongest post-hoc competitor is generate-then-prune with Covariance Matching. It generates the full pool, then keeps the same final number of queries as our method. We report it on two axes: quality at equal training-set size, and cost in total generation calls. Beating it means matching its nDCG@10 while issuing far fewer calls, not necessarily exceeding it.

For classification, the strongest baseline is AttrPrompt at its own budget. We also test AttrPrompt's attribute pool used as our seed set, which tests whether geometry adds to discrete attributes rather than replacing them.

Ablations isolate the key ingredients:

  • removing the attraction term (pure repulsion), which tests the covariance concern;
  • removing the online correction (surrogate only);
  • sweeping $s$;
  • removing the virtual antipodes.

Every student is trained with an identical recipe and tuning budget, with the selection hyperparameters chosen on a single held-out BEIR dataset.

Why not just…?

Why not just use a cheap 3B generator on everything? That is Promptodile, and it is a real threat: if calls are cheap enough, saving them barely matters. Our answer is to plot nDCG@10 against calls. If our curve reaches the full-corpus number at a fraction of the calls, the savings compound on large corpora, multi-query generation, and API-priced teachers. If random subsets already reach the same plateau early, the problem is not real for retrieval and we will say so. AttrPrompt suggests the plateau is driven by redundancy rather than generator quality.

Why not just use k-center or coreset selection? Max-min selection is the $s\to\infty$ member of our family and ignores where the corpus mass lies. It tends to over-pick outlier documents. The $s$ sweep and the attraction ablation decide this directly. If k-center ties our method, the contribution shrinks to "select before generating", which is still worth reporting, and we will not claim the energy is needed.

Why not just generate everything and prune with Covariance Matching? Pruning lowers training cost, not generation cost, and the generation cost is what we target. It may still give a better final set. The equal-size comparison shows how much quality pre-generation selection gives up in exchange for its savings.

Why not just use AttrPrompt? AttrPrompt needs designed attributes and has no counterpart for retrieval over an arbitrary corpus. On the classification datasets, the hybrid baseline tests whether geometric allocation over its attribute pool helps further.

Proposed Experiments

Retrieval runs on the BEIR datasets used by Promptagator and Promptodile. We generate with Llama-3.2-3B at budgets of 5%, 10%, 25%, 50%, and 100% of full-corpus calls, train the same dual encoder, and report average nDCG@10. Classification runs on NYT, Amazon, Reddit, and StackExchange with macro-F1, using the AttrPrompt protocol at 1%, 5%, and 20% of its simple-prompting budget.

The decisive check has two parts:

  • whether our method at 10–25% of calls reaches within noise of full-corpus Promptodile on average nDCG@10;
  • whether it beats k-center at equal calls.

The second check that could kill the idea is surrogate fidelity. If the predicted and actual output embeddings correlate poorly, and the online-corrected variant does no better than random allocation, then outputs are not predictable from seeds and steering before generation is unfounded. We also report total wall-clock, including selection, so that a claimed saving survives an honest accounting.