Reinforcement learning from verifiable rewards trains a language model with an automatic checker that says right or wrong, such as a unit test or an exact-match answer. It works well, but it has two costs. It only applies where such a checker exists, and it rewrites the model's weights. For grounded question answering, where a model reads retrieved documents, cites its evidence, and refuses when the evidence is missing, recent systems pay the second cost in full. They fine-tune the entire answering model with reinforcement learning. This proposal takes a different route. We keep the answering model (the "reader") frozen and train only a small policy that assembles what goes into the reader's prompt. The policy picks which retrieved passages to include. It also picks which worked examples of good behavior, such as a correct citation or an honest refusal, to show the reader in context. It then decides the order of both.
The bet is that the decisions which most need learning are also the ones cheapest to check. Whether a selected passage is gold evidence can be verified against annotations. Whether the reader then answered correctly, cited correctly, or correctly abstained can be verified by exact match and gating rules. In-context learning means a frozen model adapts to a task from examples placed in its prompt, with no weight updates. It supplies the "how to behave" signal that full fine-tuning would otherwise burn into the weights. If this works, grounding and honest abstention become a property of the context rather than the model. The trained policy could then be reused across readers, and the reader's other abilities would stay intact.
Core Claim
A small policy of roughly 200–400M parameters, trained online against a frozen 4B–8B reader, can recover most of the gain that full-parameter grounding RL achieves on citation and abstention. It does so while leaving the reader's general and few-shot abilities untouched, and it transfers to unseen readers. The policy needs two things. First, it must select both evidence passages and behavior demonstrations. Second, it must be trained with a gated verifiable reward that also includes a dense evidence-level term. The new element is the joint choice of demonstrations and evidence: demonstrations carry the behavior (cite, abstain) and evidence carries the content, and both are optimized by the same checkable reward.
Where This Sits
Three lines of work motivate the design. GRACE trains a reader end to end with a gated reward that runs from format, to the choice of answering versus abstaining, to content accuracy. It raises Qwen3-4B's QASPER unanswerable score from 42.23% to 74.90%. LongRLVR shows that rewarding only the final answer gives almost no learning signal when the correct evidence is sparse. It adds a dense chunk-level F-score reward. Both methods update every weight of the reader. We borrow GRACE's gating and LongRLVR's dense evidence reward, but we apply them to a context-assembly policy instead of the reader.
The closest prior work also trains something other than the reader. EviOmni trains a generative "reason-then-extract" filter with verifiable answer, length, and format rewards, and it compresses context 38.1x on Natural Questions. It handles only evidence, rewrites that evidence as generated text, and does not target abstention. MemCtrl keeps Llama 3.1 8B frozen and trains a 200M controller over an external memory using abstract-syntax-tree execution rewards. It reaches 67.4% on BFCL-v3, against 60.8% for full fine-tuning and 55.2% for plain in-context learning, and it transfers to Qwen 2.5 7B with "only a 2.6-point drop". MemCtrl provides the existence proof that a frozen backbone plus a small RL controller can beat full fine-tuning. It is, however, limited to API calls with fixed schemas. R-Mix treats demonstration selection as a sequential decision problem with outcome and process rewards, but only on sentiment classification with a static pool.
Our proposal differs from EviOmni by selecting demonstrations as well as evidence, and by targeting abstention. It differs from R-Mix by working in open-domain grounded generation with retrieval. It differs from MemCtrl by working over unstructured documents rather than tool schemas.
Two findings constrain the design. R-Mix reports that performance degrades beyond roughly eight demonstrations. VIF-RAG shows that retrieved passages distract models from following instructions. Together these mean the policy must work under a hard token budget, and it must be free to include fewer items rather than more. In-Context Learning Creates Task Vectors shows that a model compresses its prompt demonstrations into a compact internal representation that can override the rest of the prompt. This supports the premise that a few well-chosen demonstrations can set the reader's behavioral mode even when many passages sit alongside them. That paper tested only single-token tasks, so whether the premise extends to citation and abstention is part of what we test.
Research Program
We would build the pieces in the following order.
The first piece is the policy. It is a small encoder that scores candidate passages (from a standard retriever's top 50) and candidate demonstrations (from a pool) against the query. It then builds the prompt one pick at a time, choosing an item or a stop action, under a fixed token budget. Each pick is conditioned on the items already chosen.
The second piece is the demonstration pool. It is built from the training split and contains worked examples in the target format. Some demonstrations answer with sentence-level citations. Others refuse because the passages shown alongside them lack the answer. Each demonstration is tagged as either an answer or a refusal, but the policy sees only its text.
The third piece is training. We use group-relative policy optimization (GRPO): for each query, we sample several context assemblies, run the frozen reader on each, and compare their rewards within the group. The reward follows GRACE's gating:
$$ r = g_{\text{fmt}} \cdot g_{\text{path}} \cdot \big(\alpha\, \mathrm{Acc} + \beta\, \mathrm{Cite}\big) + \lambda\, F_1^{\text{ev}}, $$
The terms are defined as follows. $g_{\text{fmt}}$ and $g_{\text{path}}$ are 0/1 gates: the first checks output format, and the second checks whether the reader correctly chose between answering and abstaining. $\mathrm{Acc}$ is exact-match or F1 answer accuracy. $\mathrm{Cite}$ is citation correctness against gold evidence sentences. $F_1^{\text{ev}}$ is the F-score between the selected passages and the gold supporting passages. $\alpha, \beta, \lambda$ are fixed weights.
The dense term $F_1^{\text{ev}}$ sits outside the gates on purpose. It is LongRLVR's remedy for the near-zero probability of stumbling on the right evidence by chance, and it keeps a learning signal alive when the reader fails.
The fourth piece is analysis. We probe the reader's intermediate-layer representation at the end of the demonstration block, following the task-vector method. The question is whether "refusal" demonstrations and "citation" demonstrations produce separable internal states. If the policy has learned to steer behavior through demonstrations, then the demonstration mix it chooses should predict which of these states the reader enters.
Baselines
The naive reference point is the frozen reader with the retriever's top-k passages in retrieval order and no demonstrations. We run it at the same token budget as the policy. Its QASPER-unanswerable figure for Qwen3-4B should reproduce roughly GRACE's reported base of 42.23%.
The training-free competitor is a cross-encoder reranker for passages, plus nearest-neighbor retrieval of demonstrations by embedding similarity, packed to the same budget. It is the most likely simple answer. We tune its k values on the same validation split that the policy uses for model selection.
The strongest existing method is GRACE's full-parameter reinforcement learning of the reader. GRACE reports 74.36% overall accuracy on QASPER and 81.20% on HotpotQA with Qwen3-4B, trained on 2,000 samples. We use the same 2,000 training samples and the same readers (Qwen3-4B and Llama3.1-8B-Instruct). We match the number of rollouts per query and the total count of reader forward passes, and we give each method an equal hyperparameter search budget. Our policy "beats" GRACE in either of two cases:
- It comes within two points of GRACE on overall accuracy while strictly exceeding it on the preservation and transfer checks below.
- It matches or exceeds GRACE outright.
The closest structural competitor is an evidence-only controller in EviOmni's spirit. This is our policy restricted to passage selection, with demonstrations fixed.
Four ablations isolate the proposal's key ingredients:
- A demonstration-only policy in R-Mix's spirit, with passages fixed at reranked top-k. This isolates the value of joint selection.
- An answer-only reward with $\lambda = 0$. This tests whether the vanishing-signal problem LongRLVR identified reappears for a context policy.
- The same policy trained by supervised learning on oracle selections (gold passages plus demonstrations whose type matches the query). This tests whether reinforcement learning is needed at all.
- A shuffled-order control. This measures how much of the gain comes from ordering rather than from selection.
Why not just…?
Why not just fine-tune the reader with GRACE? This is the real threat, and it may well win on raw in-domain accuracy. Our argument rests on three measurable costs that no paper in our file quantifies for grounding RL:
- Loss of other abilities. MemCtrl's finding that weight updates hurt few-shot generalization suggests this cost exists, but no reported result was found on Lacuna for GRACE's effect on out-of-task ability.
- The need to retrain for every new reader.
- Inapplicability to readers whose weights are unavailable.
Our experiments measure all three directly. If GRACE matches us on preservation too, the case for the policy reduces to transfer alone.
Why not just use a reranker and nearest-neighbor demonstrations? Similarity has two known weaknesses:
- It cannot see whether a passage actually lacks the answer. EviOmni documents incompleteness and irrelevance in raw top-k context.
- Similar demonstrations need not teach refusal. MemCtrl notes that failed and successful traces look alike in embedding space.
Still, this baseline could match us on answerable questions. The decisive comparison is on unanswerable ones, where choosing a refusal demonstration requires recognizing missing evidence. If the reranker baseline ties there too, the reinforcement learning component is unnecessary and we would report that.
Why not just train the selector by supervised learning on gold evidence labels? This is cheap, and on HotpotQA it may be enough. But the gold labels say nothing about which demonstrations or which order actually change this reader's behavior. Evidence from other settings points the same way. RLVRR found that 10k RL examples beat 100k supervised examples in open-ended generation. ReasonRAG found that 5,000 process-supervised instances beat Search-R1 trained on 90,000. Our supervised-controller ablation settles the question for this setting.
Proposed Experiments
The main evaluation uses the following benchmarks:
- QASPER and HotpotQA. Both have gold evidence, which makes $F_1^{\text{ev}}$ and $\mathrm{Cite}$ computable. We report overall accuracy and QASPER's unanswerable score, matching GRACE's protocol.
- FollowRAG, to test whether the chosen context also keeps the reader following instructions under distracting passages.
- Transfer targets: MuSiQue and 2WikiMultihopQA as unseen multi-hop sets, which EviOmni also used.
The first decisive check is preservation. After training, we evaluate each reader on the 18 few-shot tasks from the task-vector study and on GSM8K. Our frozen reader is unchanged by construction, so the quantity measured is how much GRACE-trained readers drop. No baseline figure exists on Lacuna for grounding-RL readers on these tasks.
The second decisive check is transfer. We apply a policy trained against Qwen3-4B, without retraining, to Llama3.1-8B-Instruct and to Qwen3-14B. MemCtrl's 2.6-point cross-backbone drop sets the bar we aim for.
The third decisive check tests the dense evidence reward. Following LongRLVR, the answer-only policy should learn markedly slower, or plateau, on queries whose gold evidence is ranked beyond the retriever's top 10.
The fourth check is the probing analysis. It should show that refusal-heavy demonstration mixes shift the reader's end-of-demonstration representation toward a separable "abstain" state. If they do not, the demonstrations are helping for some other reason, and we would say so.
A clear failure outcome would be that the joint policy is no better than the evidence-only controller on unanswerable questions. That would mean demonstrations carry no behavioral signal beyond what evidence selection already provides.
References
- GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence
- LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
- Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation
- GRPO-Trained Memory Policies for Few-Shot Function Calling
- Reward Mixology: Crafting Hybrid Signals for Reinforcement Learning Driven In-Context Learning
- Toward Verifiable Instruction-Following Alignment for Retrieval Augmented Generation
- In-Context Learning Creates Task Vectors