Teaching a language model to improve itself through written critique is expensive. Methods like Self-Refinement Tuning have the model write a draft, write a critique of that draft, and write a revised answer. The model is then trained to prefer the revision over the draft. Every critique and revision is a long, strictly sequential generation, so building the training set spends most of its compute on refinement loops. Today those loops run on every prompt, whether or not the resulting draft-versus-revision pair will change the model at all. This proposal asks whether a model can decide, before paying for a full refinement, which prompts are worth refining. It borrows a tool from impact-based active learning, a family of methods that pick the training examples expected to change the model the most rather than the ones it is most unsure about.
The concrete bet is a score that runs after a cheap draft and a short capped "preview" of a revision. The score estimates how much a full refinement pair would move the model's outputs on a held-out probe set. Full critique-and-revise loops then run only on the top-scoring prompts. If the score is good, the cost of refinement is paid offline and selectively. The model that comes out should answer well in a single pass, with no refinement at deployment. The question is whether this beats simpler ways of spending the same compute.
Core Claim
At matched total compute, with selection overhead counted, impact-scored selection of which prompts to refine reaches the same downstream quality as uniform self-refinement training using a fraction of the refinement budget. It also beats random, uncertainty-based, and self-critique-based selection.
The new part is where the impact estimate sits. It acts on a refinement that has not yet happened, and it is used to cut refinement cost, not labeling cost. The closest existing work, ActiveDPO, does something different. It selects among already-generated response pairs using low-rank-adapter gradients projected to 8,192 dimensions, so that human labelers judge the most informative pairs. It treats responses as finished strings and does nothing about the cost of producing them. Active Learning for Direct Preference Optimization likewise prunes existing preference pairs, using D-optimal design on a linearized last layer. On the refinement side, Teaching Language Models to Self-Improve by Learning from Language Feedback (SRT) and SELF refine prompts uniformly and filter afterwards with scalar self-critique. SRT's authors name multi-step generation cost as an open limitation.
Research Program
For each unlabeled prompt $x$, the current model (the "policy") produces a draft $y_0$. The draft is cheap and needed anyway. It then produces $m$ capped previews $\hat y_1,\dots,\hat y_m$, each a short critique plus a revision cut off at a small token budget. Each preview is treated as a hypothetical winner over the draft, as in SRT's second stage. This is the look-ahead step of expected-model-change methods: an expectation over hypothetical outcomes replaces the unknown true outcome.
Let $g_j$ be the gradient of the preference-training loss for the pair ($\hat y_j$ preferred over $y_0$). It is taken with respect to low-rank adapter weights and compressed by a fixed random projection, following ActiveDPO. Let $\bar g$ be the mean projected gradient of log-likelihood for reference-quality answers on a small held-out probe set of prompts. The score is
$$S(x) = \frac{1}{m}\sum_{j=1}^{m} \langle g_j, \bar g \rangle ,$$
which is a first-order estimate of how much training on the pair would raise the likelihood of good answers on the probe set. This makes it an output-change score in the spirit of Expected Model Output Change. That work showed that measuring change in predictions across the input distribution, not parameter change alone, protects against outliers that look informative. A plain gradient-length variant, $\frac{1}{m}\sum_j \|g_j\|$, is the classical Expected Gradient Length and serves as an ablation. The preference loss's own sigmoid weight already down-weights pairs the model has learned. A greedy diversity penalty on the projected gradients, borrowed from ActiveDPO, prevents picking many near-duplicate prompts. Selected prompts then receive full refinement, and preference training proceeds exactly as in SRT.
The work has three parts. First, we validate the estimator by correlating $S(x)$ against the realized probe-set change measured after full refinement on a calibration subset. This is the core risk: capped previews may not predict full revisions. Second, we account for compute in floating-point operations and wall-clock GPU hours, covering drafts, previews, backward passes, and full refinements. This makes the unmeasured trade-off named in the research file explicit. Third, we study length handling, which is an open conflict in the literature. ActiveDPO found raw gradients penalize long responses and normalized them. Settles and Craven found that length normalization hurt in sequence labeling. Refinement pairs mix critique tokens with answer tokens, so we compare raw, length-normalized, and answer-tokens-only gradients.
Baselines
The naive reference point is the base model with no refinement training. Just above it is uniform SRT, which refines every prompt in the pool. This is the full-compute target. SRT reports average gains of 3.7 to 4.0 points across GSM8K, Big-Bench-Hard, and TydiQA, and lifts Tulu2-70B's AlpacaEval 2.0 win rate from 9.6% to 25.8% using 22K GPT-4 annotated instances (SRT). We work at 7B scale, so those numbers bound expectations rather than serve as direct comparison points. We rerun uniform SRT ourselves on identical prompts.
The strongest selection competitor, given equal refinement budget, is random selection at matched total compute. The selection overhead is given back to random selection as extra refined prompts. Next comes uncertainty selection, which refines the prompts whose drafts have the highest mean token entropy. Then comes SELF-style self-critique filtering, which refines prompts whose drafts the model's own short critique rates as worst; this is close to SELF's heuristic filtering moved before refinement.
We also include ActiveDPO-after-refinement. It refines everything, then keeps the top pairs by ActiveDPO's gradient score. It cannot save refinement compute, so it is an upper bound on selection quality, not a cost competitor. If our pre-refinement score approaches it, previews are capturing what full refinements would show.
Ablations isolate the key ingredient. Gradient length against probe-aligned output change tests the outlier argument. Previews against draft-only scoring, with no preview and gradients computed from the draft alone, tests whether the look-ahead is needed. The three length treatments settle the length question.
Every method uses the same prompt pool, the same 7B policy, the same refinement procedure and preference-training hyperparameters, and the same total-compute ladder (25%, 50%, and 100% of uniform SRT's cost). Hyperparameter search budgets are equal across methods. "Beating" a baseline means a higher downstream score at the same compute on at least three of the four benchmarks, with seeds' confidence intervals not overlapping. Equivalently, it means matching uniform SRT at no more than half its compute.
Why not just…?
Why not just refine a random subset? This is the real threat. If refinement gains are spread evenly across prompts, random selection with the overhead given back will win. Active learning's record here is encouraging but not decisive for this setting. EMOC beat random selection on ImageNet binary tasks with an average AUC of 80.03% against 76.83% (Freytag et al.). Kveton et al. matched uniform sampling with 4x to 8x less data in synthetic experiments (ADPO). Neither paper counts the cost of a preview. The matched-compute ladder settles this directly, and we will report a loss to random selection if it happens.
Why not just refine where the model is least confident? Uncertainty tends to fire on noisy or unresolvable prompts. On some of these, refinement produces pairs that teach little. Freytag et al. and Käding et al. (regression EMOC) document this outlier problem. ActiveDPO reports that an uncertainty heuristic, APLP, can fall below random selection (Llama-2 on WebGPT). The entropy baseline tests whether this transfers to choosing what to refine.
Why not just filter by the model's own critique score? This is cheap and close to what SELF does, but it measures how bad the draft looks, not how much fixing it will teach. A draft can be bad in a way the model already "knows," so the preference gradient is small, or bad in a way that does not generalize to other prompts. The comparison against self-critique filtering, and the draft-only ablation, show whether the gradient and probe alignment add anything over the critique text.
Why not skip training-time refinement and cut test-time refinement instead, with early stopping or internalized reflection? REFRAIN cuts reasoning tokens by 20% to 55%, and Meta-Reflection reaches single-pass reflection, at 63.4% pass@1 on MBPP. These are complementary, not competing. Both still need a refinement-capable model and its training cost, and Meta-Reflection needs layer surgery. Our output is an ordinary model whose single-pass answers improve, so it can be combined with either.
Proposed Experiments
We use Tulu2-7B on SRT's evaluation suite (AlpacaEval 2.0, GSM8K, Big-Bench-Hard, TydiQA), with a 20K-prompt pool and a fixed open mentor model for critiques. The first decisive check is the quality-versus-compute curve. The proposal succeeds only if impact selection sits above both random selection and self-critique filtering at the 25% and 50% budgets, and reaches uniform SRT's average score at no more than 50% of its compute.
The second check is the estimator correlation on a 1K-prompt calibration set. If the rank correlation between $S(x)$ and realized probe-set change is weak, and draft-only scoring performs as well, the look-ahead claim fails.
The third check is the length ablation, which reports which of the Settles–Craven and ActiveDPO findings holds for mixed critique-and-answer traces.
We also report wall-clock latency of data generation and single-pass inference latency of the trained model. If benefits appear only on reasoning benchmarks (GSM8K, Big-Bench-Hard) and not on AlpacaEval, we will treat the method as domain-specific rather than general.
References
- ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
- Active Learning for Direct Preference Optimization
- Teaching Language Models to Self-Improve by Learning from Language Feedback
- SELF: Self-Evolution with Language Feedback
- Selecting Influential Examples: Active Learning with Expected Model Output Changes
- An Analysis of Active Learning Strategies for Sequence Labeling Tasks
- Active Learning for Regression Tasks with Expected Model Output Changes
- Stop When Enough: Adaptive Early-Stopping for Chain-of-Thought Reasoning
- Meta-Reflection: A Feedback-Free Reflection Learning Framework