Attention Temperature as a Yield Control for Synthetic Query Generation

Many pipelines that train small retrievers on synthetic data follow the same recipe. A large language model writes a search query for each document. A filter then throws away every query that fails a check, and in pipelines like Promptagator the check is whether the query, fed back to a retriever, recovers its own source document. The filter often discards half or more of what the generator produced, so a large share of the generation compute buys nothing. This note proposes changing how the generator reads the document while it writes, so that more of its output survives the filter.

The control is the attention temperature: the scale applied to query-key scores inside attention, which decides whether a token focuses on a few positions or spreads over many. The proposal adjusts this temperature at every decoding step and only over the document part of the prompt, steering it toward the attention patterns that a small calibration set shows are associated with accepted queries. It is training-free and runs on an open model. The claim is narrow and testable. At a fixed total budget for generation plus filtering, a closed-loop temperature controller should yield more accepted queries, and better student retrievers on BEIR, than output-sampling temperature, a fixed global attention rescaling, or entropy-based early abort.

Why this is open, and what it must resolve

Two directions meet here. Adaptive attention temperature replaces the fixed $1/\sqrt{d}$ scale with a learned or input-dependent one. Prior work in that direction supplies the method. SACT learns a per-step temperature in RNN translation and beats every fixed temperature from 0.8 to 1.2, with content words getting sharper attention than function words. Selective Attention (SSA) learns per-query inverse temperatures during pretraining. The ICML 2026 analysis of in-context learning derives a training-free moment-ratio heuristic for setting temperature at inference and shows on Llama2-7B that the best temperature rises from 0.9 to 1.2 as demonstration noise grows. The first two require training the generator. The third only adjusts how a model consumes noisy prompts and never touches data production. I borrow the inference-time, training-free stance of the third and the per-step granularity of the first.

The closest existing work is PLATE. It softens a BART teacher's cross-attention with one fixed multiplier, $\lambda \in [1.5, 2.0]$, when producing summarization pseudo-labels, which reduces copy bias and leading bias and yields students that beat their teacher. PLATE motivates this proposal: attention temperature in the generator changes the quality of synthetic labels. It differs in three ways. Its temperature is static and global, it covers only summarization, and it never measures rejection rates or generation cost.

CARE constrains the design by pulling the other way. It argues that dispersed attention causes bland, generic text, and it sharpens attention through an entropy penalty. In query generation these correspond to two different filter failures. Copied queries come from attention that is too sharp, which is PLATE's diagnosis. Generic queries match many documents and so fail round-trip retrieval, which follows from attention that is too diffuse, CARE's diagnosis.

No fixed direction can fix both failures. That is the concrete reason to make the temperature adaptive rather than pick a side.

Core Claim

During generation, the entropy of the generator's attention over the document span predicts whether a query will survive the round-trip filter. It does so beyond what output-token entropy already tells us. Regulating that entropy per step, by rescaling only the document-span attention scores, raises filter yield and student quality at matched compute.

Concretely, at step $t$ the attention weight on document position $j$ in a selected head is proportional to $\exp\big(q_t \cdot k_j / (\tau_t \sqrt{d})\big)$. Here $q_t$ is the current query vector, $k_j$ the key at position $j$, $d$ the head dimension, and $\tau_t$ the step temperature. Positions outside the document span keep $\tau = 1$. The controller picks $\tau_t$ so that the document-span attention entropy $H_t$ stays inside a target band $[H_{lo}, H_{hi}]$. The band is estimated from about 500 calibration generations labeled accepted or rejected by the filter. The moment-ratio heuristic of the ICML 2026 paper gives the initial $\tau_t$, and one or two bisection steps on the logits refine it.

Research Program

The work proceeds in three stages, and the first stage can kill the project.

The first stage is diagnostic. Use Flan-T5 XXL, as in UDAPDR, because it has explicit cross-attention as in PLATE's setting. Use Llama2-7B as the decoder-only case, where the document sits in self-attention over the prompt. Generate few-shot queries Promptagator-style for BEIR corpora and run the round-trip filter on all of them. Record per-head document-span attention entropy during each generation.

Then ask whether these entropies predict filter outcome better than mean output-token entropy and query length do. The test is held-out AUC from a logistic probe. Also label rejected queries as copy-like or generic, using lexical overlap with the source and the number of documents they retrieve above a threshold. This tells us whether the two failure modes sit at opposite ends of the entropy axis, as the PLATE–CARE tension predicts.

If attention entropy adds no predictive value over output entropy, the steering hypothesis is dead. The note would then report a negative result on a fair test of the PLATE–CARE question.

The second stage builds the controller. It selects the heads whose entropy is most predictive, fits the band, and applies per-step rescaling. Entropy over the document span needs the attention weights for those heads only, so the added cost is one extra softmax per selected head per step. I will report it in measured FLOPs and wall-clock rather than assume it is negligible.

The third stage measures the full pipeline under a fixed total budget, counting generator FLOPs plus filter FLOPs. That budget is spent until it runs out, and a 110M dual-encoder student is trained on whatever survives.

Baselines

The naive reference point is the default pipeline. It uses Promptagator-style few-shot prompting with standard attention, the generator's default decoding settings, and round-trip filtering, run on the same generators. For context only, Promptagator reports 47.8 average nDCG@10 for its few-shot setting over 11 BEIR datasets, using FLAN 137B. Our generators are far smaller, so we reproduce the pipeline in our own setup rather than compare against that figure.

The strongest existing method is GENERATE-PAIRWISE. It generates relevant and irrelevant queries jointly and applies an LLM round-trip filter, and it reports an average +8.6 NDCG@10 over standard query generation across 5 of 6 BEIR datasets. We run it with the same generator, the same filter where applicable, and the same total compute. Our controller can also be applied inside it, so beating it can mean either outperforming it alone or improving it when combined.

On LoTTE, the reference result is UDAPDR's Lifestyle Success@5, which rises from 64.5 to 74.8. We use LoTTE as a second test with the UDAPDR student recipe. No reported result found on Lacuna for any attention-temperature method on BEIR or LoTTE query generation.

Two ablations isolate the key ingredient. The first is monitoring only: compute the same entropy signal and abort generations predicted to fail, without steering. This separates "attention entropy is a useful cheap pre-filter" from "changing temperature helps." The second is a fixed document-span temperature, PLATE's design moved to query generation, with $\lambda$ swept over the same number of configurations the controller uses to fit its band.

Every method gets eight tuning configurations, selected on a held-out dev split of one corpus (SciFact), and the same compute ceiling. Beating a baseline means one of two outcomes. Either the student's nDCG@10, averaged over the evaluation corpora, is higher at equal budget with a bootstrap interval excluding zero, or equal nDCG@10 is reached with at least 30% less total compute.

Why not just…?

Why not just tune output sampling temperature and top-p? This is the most serious competitor, because it is free, works through any API, and plainly changes acceptance rates. CARE's argument is that generic text comes from hidden states that have already collapsed toward common-word embeddings, so resampling at the output cannot restore focus on informative document tokens. That argument comes from open-ended generation, not query synthesis. We therefore sweep output temperature with the same eight-configuration budget and also stack it with our controller. If output temperature alone matches yield and student quality, we report that attention steering is unnecessary for this purpose.

Why not just use one fixed attention rescaling, as PLATE did? SACT beat every fixed temperature in translation, but that was an RNN result and may not transfer. The stronger reason is the two-sided failure mode. If the stage-one diagnostic shows that copy-like and generic rejections sit at opposite entropy extremes, any single $\lambda$ must trade one failure for the other. The fixed-$\lambda$ ablation tests exactly this. If the rejections cluster on one side instead, the fixed scale may suffice, and the adaptive part would not be justified.

Why not just over-generate and filter with a cheaper model? Cheaper filters lower cost per candidate but do not cut waste in the generator itself. Promptagator also reports that its filter was too strict on small corpora such as NFCorpus, which suggests the bottleneck is what gets generated, not only the price of checking it. The monitoring-only ablation is in effect a near-free filter applied mid-generation, so the experiments directly measure how much of any gain comes from cheap rejection versus better generation.

Why not just fine-tune the generator toward filter acceptance, as ScaleQuest does with preference optimization? This is a real threat at large scale. ScaleQuest shows that a tuned 7B generator can produce 1M problems for about $680, and training costs are paid once and spread over every sample. We run a preference-optimized generator trained on the calibration-set filter labels and include its training compute in the budget. We report the corpus size at which it overtakes the training-free controller, which is the honest version of the comparison.

Decisive Checks

Evaluation uses BEIR's ArguAna, FEVER, Climate-FEVER, SciFact, NFCorpus, TREC-COVID and Touché-2020, plus LoTTE Lifestyle and Technology. The primary metrics are nDCG@10 and Success@5. Cost is reported as total FLOPs and wall-clock per accepted query.

Because round-trip filters favor queries that copy the document, every accepted set is also reported with source n-gram overlap and the distinct-n diversity used by CARE. A controller that raises yield by producing copy-heavy queries counts as a failure, even if acceptance rises.

FEVER gets separate analysis because entity-centric hard negatives are where the file reports zero-shot generation is weakest. There we test whether sharper document attention on entity tokens raises yield specifically.

As a sanity anchor, the controller is applied to BART summarization pseudo-labeling on CNN/DailyMail. It should recover at least PLATE's gain over standard pseudo-labels without hand-picking $\lambda$. Failing that check would mean the controller is not doing what PLATE's fixed scale does, and the retrieval results would need a different explanation.