Reinforcement learning (RL) can make reasoning models less prone to stating false facts, but only if the training reward actually checks the facts. KnowRL, which trains a reasoning model with a reward that splits each reasoning chain into short claims and looks each one up in Wikipedia, cut the incorrect-answer rate of DeepSeek-R1-Distill-Qwen-7B on SimpleQA from 78.00% to 57.67% KnowRL. Its authors also identify the cost: retrieving evidence and splitting claims for every sampled reasoning chain during online training is slow. Most of those checks are probably wasted. Many chains are obviously right or obviously wrong from cheap signals, and checking them changes nothing about the gradient step. A smaller set of chains look fine on the surface and hide a fabricated fact, and those are exactly the ones a cheap reward mis-scores.
We propose a small learned scheduler that sits inside the RL loop. For each batch of sampled reasoning chains it decides two things: which few chains are worth sending to the expensive fact-checker, and which unchecked chains are risky enough to be dropped from the update entirely. The scheduler is a contextual bandit, a learner that repeatedly picks among options using features of the current situation and observes a reward only for the options it picked. Its reward is how much the fact-check changed that chain's training signal. The claim is that a model trained with fact-checks on a fraction of its chains, allocated this way, keeps most of KnowRL's factuality gain at a fraction of the retrieval cost. It should also beat the obvious ways of allocating the same budget: at random, or by ranking chains on model uncertainty.
Core Claim
The value of verifying a rollout (one sampled reasoning chain and answer) is concentrated: verification matters only when it would flip or substantially move that rollout's advantage, meaning its score relative to the other rollouts for the same prompt in Group Relative Policy Optimization (GRPO). This value can be predicted from cheap features, and the prediction target is observable right after each check. That makes the scheduler's reward immediate, which avoids the delayed credit-assignment problem that makes RL data selection noisy elsewhere A Snapshot of Influence. Concretely, for rollout $i$ in a group, let $A_i^{\text{cheap}}$ be its advantage under the outcome-plus-format reward alone and $A_i^{\text{full}}$ its advantage once the claim-level factuality reward is added. The scheduler, which picks rollouts to verify under a budget of $k$ checks per group, receives
$$ r_i = \left| A_i^{\text{full}} - A_i^{\text{cheap}} \right| - \lambda \, c_i , $$
where $c_i$ is the retrieval cost of checking rollout $i$, measured in claims looked up, and $\lambda$ trades cost against value. Unverified rollouts whose predicted $r_i$ is high, meaning the scheduler suspects a hidden fabrication it could not afford to check, are hard-gated out of the policy update instead of being scored by outcome alone.
Relation to Prior Work
Two papers supply the parts, and the gap lies between them. The closest work on selection is the Contextual Bandit Scheduler of Lu et al.. It scores each rollout with a ten-number summary of training dynamics (reward, advantage, length, sample age, and so on), prunes low-value rollouts within a group, and replays useful old ones. It reports 50.4% less optimization time and up to 8.5% higher average scores on math benchmarks. We reuse its bandit-over-rollouts structure and its finding that the scheduler needs an entropy penalty, or it learns to favor repetitive text. We differ in three ways. Our domain is factual question answering, where no rule-based checker exists. The scheduler's main action is to allocate verification, not just to keep or drop. And its reward is the change verification causes, not the policy gain between optimization steps, which is noisier. KnowRL supplies the verifier and the composite reward. It applies them to every rollout, which is the cost we attack. Gap 3 of the research file names this combination; we found no paper that tests it.
Several papers motivate the design. Wu et al. show that rewards of +1 for correct and −1 for incorrect teach models to guess. This is why unverified rollouts must not simply fall back to outcome-only scoring: a lucky guess backed by a fabricated chain would be rewarded. Their calibrated confidence and critic-value signals also become cheap scheduler features. Agrawal et al. show that a model's answers vary when it is asked indirectly about its own fabrications. That suggests a cheap feature: disagreement among the rollouts for the same prompt on their stated claims. Hard gating instead of soft down-weighting follows the argument from distant-supervision denoising Feng et al.; Zeng et al.. Soft weights still pass some gradient from corrupted examples, while a zero weight removes them.
Other work constrains the design. Policy Filtration for RLHF finds that reward models are reliable at the extremes and noisy in the middle. If cheap rewards behave the same way, mid-range rollouts should carry most of the value of verification, and we test this directly. Its static, ground-truth-tuned filters are also a baseline. Qi et al. select preference pairs where the model's implied preference margin is smallest. Lin et al. show that larger models absorb popular misconceptions. Together these warn that uncertainty-only selection may skip confidently held falsehoods. A scheduler trained on verification outcomes could learn to catch those, while a rule that ranks by uncertainty could not.
Research Program
We build on KnowRL's GRPO setup with the same training prompts, Wikipedia knowledge base, claim splitter, and the two base models it reports, DeepSeek-R1-Distill-Qwen-7B and Skywork-OR1-7B. At each step and for each prompt group, every rollout receives the cheap reward and a feature vector. The vector combines Lu et al.'s ten dynamics features with four hallucination-oriented ones: mean token entropy, claim-level disagreement with sibling rollouts, verbalized confidence where the model is prompted to give it, and number of claims. A two-layer scoring network, small enough to run alongside training at negligible cost, predicts $r_i$.
The top $k$ rollouts by predicted value are verified. A further fixed 10% of the budget is spent on uniformly random rollouts. This gives exploration and an unbiased estimate of how accurate the scheduler is. Rollouts that are not verified and whose predicted value exceeds a threshold are excluded from the update. The remaining unverified rollouts keep the cheap reward. The threshold is tuned once on a held-out set of prompts. We follow Lu et al. and add an entropy term to the scheduler objective, and we monitor rollout repetition rates.
The analysis asks three questions. Is verification value in fact concentrated, so that a small share of rollouts accounts for most of the total $|A^{\text{full}} - A^{\text{cheap}}|$? We measure this first under full verification, before any scheduler is trained. Does the scheduler's ranking differ meaningfully from an entropy ranking? Do the rollouts it chooses include confident fabrications, as the imitative-falsehood concern predicts they should?
Baselines
The naive reference point is GRPO with outcome and format rewards only and no fact-checking, run on the same prompts. It shows how much factuality any verification buys.
The quality ceiling is full KnowRL with every rollout verified. Its reported SimpleQA incorrect rate for DeepSeek-R1-Distill-Qwen-7B is 57.67%, against 78.00% for the untrained model. On GPQA with Skywork-OR1-7B it reports 42.42% against 37.37% KnowRL. We will reproduce it in our own setup and compare against our reproduced number, since a match is not guaranteed. It is the strongest existing method on this question. Our success criterion against it is efficiency, not raw accuracy: recover at least 80% of its SimpleQA incorrect-rate reduction while using at most 25% of its retrieval calls.
The strongest budget-matched competitors get exactly the same number of checks per group, the same prompts, the same number of GRPO steps, and the same tuning budget for any threshold. The first is random verification. The second is static uncertainty ranking: verify the $k$ rollouts with the highest token entropy, or the highest sibling disagreement, whichever validates better. The third is CBS ported faithfully, with its original features and policy-gain reward, choosing which rollouts to verify. The scheduler beats them if its SimpleQA incorrect rate is lower at the same budget by more than the spread across three seeds, at two or more budget levels.
A behaviorally calibrated RL run in the style of Wu et al., which uses no retrieval at all, tests whether calibration alone suffices. The file reports only that its SimpleQA calibration error is on par with frontier models; no reported scalar result was found on Lacuna for its SimpleQA accuracy.
Two ablations isolate the key ingredients. One removes the hard gate, so unverified rollouts keep the cheap reward. The other replaces the verification-change reward with CBS's policy-gain reward while keeping our features.
Why not just…?
Why not just verify a random subset? Random sampling at budget $k$ is unbiased, and if verification value is spread evenly across rollouts it is the right answer. The concentration measurement in the research program settles this before any scheduler is trained. If the top 25% of rollouts carry only about 25% of the advantage change, we stop and report that random checking is enough. That would be a useful negative result for anyone running KnowRL-style training.
Why not just rank rollouts by uncertainty? This is the real threat, because entropy and sibling disagreement are free and are probably most of what the scheduler learns. The case for learning rests on two things. The cheap-reward noise pattern found by Policy Filtration for RLHF depends on reward level, not only on uncertainty. And confident misconceptions Lin et al. are low-uncertainty errors. We will report the rank correlation between the scheduler's choices and the entropy ranking, together with accuracy. If the correlation exceeds 0.9 and the accuracy gap falls within seed noise, we will conclude that the static heuristic suffices and recommend it.
Why not just use calibration rewards and skip retrieval? Wu et al. show calibration transfers from math to SimpleQA without factual training. Calibration teaches the model when to abstain, though, not which facts are true. KnowRL's gains come from checking claims against evidence. The calibrated baseline above, and a combination of calibration with our scheduler, will show whether the two are complementary or redundant.
Why not just make the verifier cheaper? A small judge model in place of retrieval would cut cost along a different axis. HaluEval reports that even ChatGPT recognizes hallucinations in QA at only 62.59% without retrieval, rising to 76.83% with it HaluEval. Retrieval-free checking is therefore weak. The scheduler also works with any verifier, so cheaper verifiers combine with it rather than replace it.
Proposed Experiments
The main result is a curve of SimpleQA incorrect rate against total retrieval calls. It is traced at budgets of 10%, 25%, 50%, and 100% of rollouts for each allocation method, on both KnowRL base models, with three seeds each. SimpleQA is scored as correct, incorrect, or abstained, so that a lower incorrect rate gained purely through more abstention is visible. Out-of-distribution truthfulness is measured on TruthfulQA, both MC1/MC2 and generation scored by GPT-judge, which agrees with human raters 90–96% of the time Lin et al.. Reasoning retention is measured on GPQA and OlympiadBench, where KnowRL reports 42.42% and 5.75% for Skywork-OR1-7B. Wall-clock training time and repetition rates are logged throughout.
The decisive checks are three. First, verification value must be concentrated under full verification. Second, at the 25% budget the scheduler must lie below both random and uncertainty-ranked allocation by more than the seed spread. Third, removing the hard gate must raise the SimpleQA incorrect rate, which would confirm Wu et al.'s guessing mechanism inside our setup. Failure on the first check ends the project with a clean negative result. Failure on the second alone means the contribution is the budgeted-verification recipe with a static ranking, which we would report as such.
References
- KnowRL: Exploring Knowledgeable Reinforcement Learning for Factuality
- A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
- Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
- Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning
- Do Language Models Know When They're Hallucinating References?
- Reinforcement Learning for Relation Classification From Noisy Data
- Large Scaled Relation Extraction With Reinforcement Learning
- Policy Filtration for RLHF to Mitigate Noise in Reward Models
- Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models