Fine-tuning a language model with reinforcement learning from a pass/fail checker is now a standard way to improve mathematical reasoning. The method is often called RLVR, reinforcement learning with verifiable rewards. Most practitioners do not update all the weights. They train small add-on modules, called adapters, such as LoRA (low-rank matrices added beside existing weight matrices). Which adapter to use, where to place it, and at what size matters a great deal. On the same 1.5B model and data, one adapter variant collapses to near-zero accuracy while another beats full fine-tuning (Evaluating Parameter Efficient Methods for RLVR).
Choosing among these designs is a small architecture-search problem, and the usual way to make search affordable breaks under sparse reward. Neural architecture search makes candidates cheap to compare by training each one briefly and trusting that the short run ranks them correctly. In RLVR a short run on hard problems mostly produces groups of samples that all fail. Those groups carry zero learning signal, so a brief run tells you almost nothing about a candidate. This note proposes a score computed once from the base model's own samples on "boundary" problems, those it solves only occasionally. The score ranks thousands of adapter designs at the cost of one backward pass each, with no RLVR training per candidate.
Core Claim
An adapter design's eventual RLVR performance is predicted by a simple quantity: how much expected reward a small update through that adapter's parameters can buy per unit of change to the model's output distribution. The quantity is measured at the starting point, using only prompts on which the base model's success rate is low but nonzero.
All candidates start from the same frozen base model, so the expensive part can be computed once and shared. That part is sampling rollouts and identifying boundary prompts. Each candidate then needs only gradients with respect to its own parameters. By contrast, a short RLVR run needs fresh rollouts for every candidate. The proposal makes three testable claims. First, this score ranks adapter designs better than short RLVR runs of equal compute. Second, it flags failing designs, such as the principal-subspace initialization that collapses in the PEFT study, before any training. Third, the design it selects matches or beats the strongest hand-picked adapter.
The proposal also uses this tool to probe a scientific question the file leaves open. Do adapter choices change true expansion, meaning solving problems the base model never solves, or only "sharpening" of what the base model already almost knows?
Why the obvious version does not work, and what prior work supplies
Three lines of work motivate the question.
- Zero-cost proxies. These are formulas that score a network at random initialization from one forward or backward pass. They reach moderate rank correlation in vision search spaces; an evolved proxy reaches Kendall tau 0.65 on NASBench-201 (EZNAS). The same paper reports that they fail to separate the top 10% of models. They are also built on assumptions about convolution blocks that do not carry over to adapters on a pretrained transformer. Our score is closer in spirit to a zero-cost proxy, but it is computed on the actual reward signal rather than on reward-blind statistics of the network.
- Budgeted performance estimation. Zheng et al. show that search needs only rank preservation, not absolute accuracy, and that small training budgets can preserve rank (Rethinking Performance Estimation in Neural Architecture Search). They also report a gap between early-training and final correlation. This paper supplies our evaluation criterion, rank preservation. It also supplies our main baseline, the low-budget run. We argue that sparse reward specifically breaks the low-budget run, because short RLVR runs are dominated by zero-advantage groups.
- Adapter choice under RLVR. The PEFT study supplies the ground-truth phenomenon. RLVR updates concentrate in low-curvature directions away from the principal subspace, and adapter choice changes outcomes by tens of points.
The score's form is borrowed from trust-region reasoning. The theory of Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training explains the restriction to boundary prompts. It shows that direct RL on deep reasoning trees needs exponentially many samples, while progress through intermediate-difficulty problems needs only polynomially many. Boundary prompts are where usable signal lives, so they are where an adapter's ability to absorb that signal should be measured.
Two methods supply mechanics we reuse. Knapsack RL allocates rollouts toward prompts likely to give nonzero gradients; we use it to find boundary prompts cheaply. Adaptable Adapters learns per-layer on/off switches for adapters, using a differentiable relaxation of discrete choices (Gumbel-Softmax) trained on a supervised loss. It defines the layer-placement search space we adopt, and it is the natural supervised-search baseline.
Research Program
The search space. We define a space of adapter designs for a reasoning model in the 1.5B range, using the DeepSeek-R1-Distill-1.5B setup of the PEFT study. Each design specifies three things:
- per-layer placement, in contiguous layer bands;
- which matrices are adapted (attention projections, MLP projections, or both);
- the adapter family (LoRA, DoRA, or principal-subspace initializations such as PiSSA and MiLoRA), plus rank.
Here DoRA means weight-decomposed low-rank adaptation, which trains magnitude and direction separately. The space is a few thousand designs, small enough that every one can be scored.
The score. We sample $N$ rollouts per prompt from the base model on DAPO-Math-17k, using Knapsack-style allocation. We keep the set $B$ of prompts whose empirical success rate is strictly between zero and a ceiling, for example 1/8. For each design $a$, the trainable adapter parameters at their standard initialization are $\phi_a$. Two quantities are computed on $B$:
- $g_a$, the gradient of the GRPO objective (the group-relative policy-gradient loss used in RLVR) with respect to $\phi_a$;
- $F_a$, the diagonal of the Fisher information, i.e. the mean squared per-token score with respect to $\phi_a$.
The score is
$$S(a) = \sum_i \frac{g_{a,i}^2}{F_{a,i} + \epsilon},$$
where $i$ indexes adapter parameters and $\epsilon$ is a small damping constant.
The form comes from a standard trust-region argument. Under a second-order approximation to the KL divergence, the best step of KL size $\delta$ through parameters $\phi_a$ improves expected reward by about $\sqrt{2\delta\, g_a^\top F_a^{-1} g_a}$. $S(a)$ is the diagonal version of $g_a^\top F_a^{-1} g_a$, so ranking designs by $S$ ranks them by reward gained per unit of policy change.
This should penalize designs that adapt high-curvature principal directions. There, any step that helps reward also moves the distribution a lot, which is a plausible mechanism for the spectral collapse the PEFT study observes.
Two variants address known weaknesses. A "refreshed" variant recomputes $S$ after a handful of RLVR steps, to test whether the step-zero gradient misses later dynamics, the local-versus-global gap of Zheng et al. A "transfer" variant adds alignment between $g_a$ and the likelihood gradient of answer-conditioned self-explanations on zero-success prompts, in the style of ExPO. It tests whether signal from the hardest problems adds predictive value.
Ground truth. We pick 24 designs stratified across the score's range, plus every hand-designed configuration from the PEFT study. Each is trained with full RLVR using one fixed recipe and three seeds. Each is evaluated on MATH-500, AIME, AMC and Minerva, and on expansion over the zero-baseline problems of MATH-Beyond. We report all costs in GPU-hours: the shared rollouts, the per-design backward passes, and each full run.
Baselines
Naive reference points. These are random design selection and ranking by trainable-parameter count. If the score does not beat parameter count, the claim fails.
Low-budget RLVR run. This is the strongest competitor for the ranking task. It is the method of Rethinking Performance Estimation in Neural Architecture Search carried over directly: train each candidate for a few RLVR steps and rank by reward. For fairness we give it exactly the total GPU-hours that scoring the full space with $S$ costs, spread over a subsample of designs it can afford. We also give it Knapsack rollout allocation, so it is not handicapped by wasting rollouts on all-fail prompts. Beating it means higher Kendall tau against final accuracy on the 24 ground-truth designs, with a gap larger than the seed-to-seed spread of tau. We also require better top-10% discrimination, the regime where EZNAS reports zero-cost proxies breaking down. No reported result found on Lacuna for rank correlation of any proxy over adapter designs under RLVR, so both numbers are new measurements. The vision reference point is EZNAS-A's 0.65 tau on NASBench-201.
Reward-blind gradient proxies. These are gradient-norm or Synflow-style scores computed on adapter parameters with a plain likelihood loss. They isolate the key ingredient, the reward signal from boundary prompts. A further ablation, $S$ computed on all prompts, including all-fail ones that contribute zero gradient, isolates the boundary-prompt restriction.
Strongest final model. This is DoRA on all layers. It reaches 46.6% average accuracy on DeepSeek-R1-Distill-1.5B, against 42.5% for standard LoRA and 44.9% for full-parameter tuning (Evaluating Parameter Efficient Methods for RLVR). We rerun it under our recipe and seeds rather than copy the number. A design chosen by $S$ beats it only if its mean over three seeds exceeds DoRA-all by more than the larger seed standard deviation, at equal or fewer trainable parameters.
Supervised differentiable search. We run Adaptable Adapters-style learnable layer switches on supervised traces, then train the discovered design with RLVR. It gets the same compute as scoring with $S$.
Why not just…?
Why not just use DoRA everywhere and skip search? This is the most serious threat. The PEFT study already finds DoRA beats full fine-tuning, and a searched design may add little. We treat this as a possible outcome, not something to argue away. Even if no searched design beats DoRA-all, the score earns its place if it ranks designs well. It would let practitioners reject collapsing designs before paying for a run, and check new adapter families or base models where no study exists. The decisive comparison is the final-model one above. If DoRA-all wins, we will say so, and the contribution narrows to the diagnostic.
Why not just run short RLVR trials, as standard budgeted search does? Our hypothesis is that on hard data short runs see mostly zero-advantage groups, so their rankings are noise. The hypothesis has a precise test. We plot the low-budget baseline's tau against the fraction of zero-advantage groups in its trials, over training subsets of increasing difficulty. If short runs rank well even on hard subsets, the zero-reward argument is wrong. The remaining case for the score would then be cost alone, from reusing shared rollouts.
Why not just fix exploration instead of the architecture, with 3PO or Knapsack allocation? These are fixes to the training algorithm. Parameter Exploration for RLVR via Variational Learning perturbs weights to rescue zero-advantage groups, and Knapsack RL reallocates rollouts. They are orthogonal to adapter choice rather than substitutes. We test this by training the top-scored design and DoRA-all with 3PO on top. If the gap between designs vanishes once exploration is improved, adapter choice matters only through exploration, which is itself a useful finding.
Why not just search with a supervised loss, as Adaptable Adapters does? The supervised gradient rewards imitating traces. RLVR rewards shifting probability toward correct outcomes with little distribution change. The PEFT study's finding, that RLVR updates live away from principal directions, suggests the two objectives favor different designs. The supervised-search baseline measures whether this difference matters in practice.
Proposed Experiments
The first decisive check is ranking. On the 24 ground-truth designs, we compare Kendall tau and top-10% precision for $S$ against the equal-compute low-budget runs, the reward-blind proxies, and parameter count. We also check whether $S$ ranks PiSSA and MiLoRA, and the very small adapter families the PEFT study found too weak for reasoning, at the bottom without training them.
The second is selection. We take the top three designs by $S$ from the full space, train each with three seeds, and compare against DoRA-all on the four standard math benchmarks.
The third is the structural-versus-algorithmic question raised by MATH-Beyond. Across the ground-truth designs, we regress MATH-Beyond expansion on average benchmark accuracy. Expansion that is flat across designs, with accuracies that differ, would be strong evidence that adapter structure only sharpens. For calibration, leading RL models in that paper reach 5.22% expansion (DeepScaleR-1.5B) and 9.57% (Nemotron-v2). Expansion that tracks $S$ beyond what accuracy explains would suggest that how efficiently a design absorbs boundary-prompt signal is linked to reaching problems the base model never solved.
Finally, we repeat the ranking check with the refreshed and transfer variants. This shows whether a few RLVR steps, or signal from zero-success problems, is needed to close any remaining gap.
References
- Evaluating Parameter Efficient Methods for RLVR
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Rethinking Performance Estimation in Neural Architecture Search
- Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
- Knapsack RL: Compute-Efficient Reinforcement Learning via Heterogeneous Rollout Allocation
- Adaptable Adapters
- ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning
- MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model
- Parameter Exploration for RLVR via Variational Learning