Audited Hop-Level Factuality Rewards for Cheaper Truthful Reasoning RL

When language models are trained with reinforcement learning to reason step by step, they tend to hallucinate more. They are rewarded for reaching the right final answer, so an invented intermediate "fact" goes unpunished as long as the answer comes out right. The most direct fix is to check every sentence of every sampled reasoning chain against external evidence during training and punish the ones that are contradicted. This works well, but it is slow: the checker runs inside the generation loop, and its authors say the cost is what stops them from scaling past 8B parameters. We propose to keep most of the benefit while calling the external checker on only a small, random fraction of reasoning steps. Two ideas make this possible. The first is to reward whole reasoning hops, meaning the span of text that commits to one new fact, rather than individual sentences. The second is a cheap learned predictor of hop factuality. The real checker audits a random subset of hops, and those audits correct the predictor's reward in a way that stays unbiased no matter how wrong the predictor is.

The two sentences a colleague should take away are these. Treat a reasoning chain as a short sequence of fact-committing hops rather than a long sequence of tokens or sentences, and learn a value predictor over those hops. Then spend the expensive external fact-check only on randomly audited hops, and combine audit and prediction so the policy cannot learn to fool the cheap predictor. If this works, factuality-aware RL becomes a matter of choosing a verifier budget instead of a fixed per-sentence tax.

Core Claim

The closest work is Reasoning Models Hallucinate More (FSPO). FSPO splits each rollout into sentences and runs a Natural Language Inference (NLI) checker, a model that decides whether a claim is entailed by, neutral to, or contradicted by evidence, against Wikipedia for every sentence. It then flips the sign of the advantage for tokens in contradicted sentences, within Group Relative Policy Optimization (GRPO), an RL method that scores each rollout against the other rollouts sampled for the same prompt. FSPO motivates this proposal in two ways. It shows that dense factual rewards lift a Qwen-2.5-7B base model from 48.0% to 83.0% accuracy on HaluEval-QA, where the reasoning-distilled R1-Distill-Qwen-7B scores only 11.6%. It also names computational scalability as its main limitation.

Our claim is specific and falsifiable. Take an audit rate $q$ (the fraction of hops sent to the real checker) of roughly 0.1 to 0.25. Use a hop-level reward estimator that combines a learned predictor with randomly audited checks. At matched training steps, this should reach HaluEval-QA and multi-hop QA factuality within a few points of full per-sentence FSPO. It should also clearly beat FSPO with the same number of checker calls spent by uniformly subsampling sentences. What is new is not "verify less." It is verifying less with an estimator that the policy cannot exploit, defined over temporally abstract units.

The estimator is where the idea lives. Let $h_k$ be the text of the chain up to the end of hop $k$. Let $v_k \in \{-1, 0, +1\}$ be the external checker's verdict on hop $k$. Let $V_\phi(h_k)$ be a small learned head that predicts that verdict. Let $m_k \in \{0,1\}$ be an audit indicator drawn with probability $q$, independently of anything the policy generates. The per-hop factuality reward is

$$ \hat r_k = V_\phi(h_k) + \frac{m_k}{q}\,\big(v_k - V_\phi(h_k)\big). $$

Averaged over the audit coin, $\hat r_k$ equals $v_k$ exactly. A policy that learns to write text the predictor scores highly gains nothing in expectation, because every audit corrects the predictor's error, scaled up by $1/q$. The predictor's only job is to reduce variance. How far $q$ can be lowered depends on how accurate $V_\phi$ is, which is why hop-level structure matters. This is the standard control-variate or "doubly robust" construction. Applying it to in-loop factuality verification for language model RL is what we have not seen in the file.

Why Temporal Abstraction Enters

Temporal abstraction means grouping many primitive steps into one higher-level step, called an option or macro-action. Approximate Planning in POMDPs with Macro-Actions showed that planning over hand-built macro-actions turns the problem into a semi-Markov decision process, a decision problem whose steps have variable duration. This speeds up how reward propagates and stabilizes value estimates. Option-aware Temporally Abstracted Value (OTA) supplies the specific method we reuse. It learns high-level values over jumps of $n$ steps with a discount that does not depend on $n$. It shows that the usual failure of long-horizon hierarchical RL is "order inconsistency," where the value estimate gets worse as the agent moves closer to the goal. Horizon contraction fixes this in navigation: OTA reaches 79% success on HumanoidMaze-giant against 0–3% for HIQL.

We carry this over to text. A "hop" is the span between two points where the chain commits to a new checkable claim, such as a new entity, date, or relation. The segmenter that finds these points is fixed and external: a small claim-boundary tagger trained on HotpotQA and 2WikiMultiHopQA evidence annotations. It is deliberately not produced by the policy, for two reasons. RA3 shows that abstractions keyed to model-emitted markers (its # comments) become brittle. And letting the policy decide its own boundaries would let it choose what gets checked.

A typical multi-hop answer contains a few hops but many sentences and hundreds of tokens. $V_\phi$ is trained on this contracted horizon, with discounting per hop. The advantage for hop $k$ is computed from $\hat r_k$ and the later hops, then assigned to that hop's tokens, following FSPO's sign-flip rule. We will test directly whether OTA's order-inconsistency pathology shows up in language: along chains whose hops are all audited as entailed, is the predicted downstream factuality value monotone?

Two further works are related but different. CoLA also abstracts the action space, replacing a 128k-token vocabulary with 64 learned latent actions, and reports MATH-500 rising from 38.2 to 42.4. However, it targets math and agent tasks and changes the model's architecture. We leave the policy's token interface untouched and abstract only the reward and value structure, which keeps the method a drop-in change to GRPO. Micro-Macro Retrieval uses "macro/micro" to mean two retrieval stages with rule-based rewards. It does not model decisions as options, so we treat it as a strong retrieval-augmented comparison point rather than a competitor for the same mechanism.

Research Program

The first component is the hop segmenter and the audit pipeline. The audit pipeline is FSPO's checker (retrieval plus NLI against Wikipedia) run on a whole hop instead of a sentence. We will measure how NLI accuracy changes as spans get longer, because longer hypotheses may weaken the checker. The second component is $V_\phi$, a lightweight head on the frozen reference model's hidden states at hop boundaries. It adds no extra full-size model to the memory budget, which matters given the four-model memory cost of standard online RL described in the file's cited overhead work. $V_\phi$ is trained online by regression on audited hops only, so it tracks the current policy's error distribution. This is the same on-policy concern that motivates APRM, which found that reward models trained on static data miss new kinds of errors. The third component is the policy update: GRPO with FSPO's sign-flip rule, driven by $\hat r_k$ instead of per-sentence verdicts. We will also clip $\hat r_k$ lightly, because the $1/q$ factor can produce large rewards at small $q$. Any bias this introduces will be reported.

The analysis traces a cost-quality curve over $q \in \{1.0, 0.5, 0.25, 0.1, 0.05\}$. Cost is measured as checker calls per rollout, wall-clock time per training step, and GPU-hours to reach a target HaluEval-QA accuracy. We will also track the gap between $V_\phi$ and audit outcomes over training, which tells us whether the policy is drifting into regions the predictor cannot model. Experiments start at Qwen-2.5-7B to match FSPO exactly. We then move to a 14B model, the scale FSPO describes as a significant computational hurdle.

Baselines

The naive reference point is outcome-only GRPO rewarded on final-answer correctness, the regime that produces FSPO's reasoning-factuality paradox. Alongside it we report the untuned base model, for which FSPO gives 48.0% HaluEval-QA accuracy on Qwen-2.5-7B, and R1-Distill-Qwen-7B at 11.6%. Beating outcome-only GRPO on factuality while matching its answer accuracy is the minimum bar.

The strongest existing method is full FSPO: per-sentence checking on every rollout, which corresponds to $q=1$ at sentence granularity. It is the strongest competitor because it is the only method in the file that directly optimizes intermediate factuality inside RL and reports large gains. It reaches 83.0% HaluEval-QA accuracy and 75.5% on MATH-500 from a Qwen-2.5-7B base, as reported in FSPO. We will re-run it ourselves on the same base model, training prompts, retrieval corpus, NLI checker, and hardware, and give it the same hyperparameter search budget as our method. No reported result was found on Lacuna for FSPO's wall-clock time or checker-call counts, so all cost comparisons come from our own runs. Beating FSPO means reaching within two points of its HaluEval-QA accuracy with at most a quarter of its checker calls, and no loss on MATH-500. At 14B, it means completing training in a budget where full FSPO cannot.

The key ablation is budget-matched FSPO. Sentences are subsampled uniformly at random at the same number of checker calls, and unaudited sentences receive zero factuality reward. This isolates whether the learned predictor and the unbiased correction add anything beyond "check less often."

Two further ablations separate the ingredients. The first uses the same estimator at sentence granularity instead of hop granularity, which tests whether horizon contraction helps the predictor. The second replaces audit-corrected rewards with $V_\phi$'s prediction alone. This learned-critic-only variant is expected to be exploited by the policy, and we want to measure how quickly that happens. For retrieval-heavy multi-hop QA, we also compare against Micro-Macro Retrieval, which reports 20.87 exact match on MuSiQue at 3B against 19.40 for the previous RL baseline. That comparison runs at 3B so the numbers are commensurate.

Why not just…?

Why not just subsample FSPO's sentence checks at random? This is the most serious threat, and it is the key ablation above. The argument against it is about variance. In a GRPO group of about eight rollouts with around ten sentences each, auditing 10% of sentences means most contradicted sentences are never penalized in a given update. The sign-flip signal becomes sparse and noisy. The predictor carries information learned on other rollouts to the unaudited hops. If budget-matched subsampling matches our method at $q=0.25$ and $q=0.1$, then the predictor adds nothing. We would report the result as a cost study showing that FSPO tolerates heavy subsampling, which is still useful, and we would drop the claim about the learned predictor.

Why not just use a fast learned process reward model instead of the external checker? A process reward model is a learned scorer for individual reasoning steps. Used alone as the reward, it is exactly the learned-critic-only ablation. APRM documents that static process reward models fail on errors outside their training distribution. Its remedy is an adversarial generator, which adds more RL cost. Our audits provide the same on-policy correction at a fraction of that cost, and they guarantee the reward cannot be exploited in expectation. The ablation will show how quickly a predictor-only reward degrades, measured as the predictor-audit gap rising and HaluEval-QA falling.

Why not just trigger checks when the model looks uncertain, or only on sentences containing entities? Both are cheap heuristics. The entity filter is partly absorbed into our hop segmenter. Uncertainty-triggered checking is dangerous if the trigger reads the policy's own confidence, because the policy can learn to sound confident and avoid checks. HaluEval shows models are poor at noticing their own errors, with ChatGPT at 58.53% detection accuracy on summarization, so self-reported confidence is a weak trigger. We will still run an entropy-triggered variant as an extra baseline. Our estimator permits a non-uniform audit rate as long as it is set without reference to the policy's output and is floored away from zero, so a hybrid remains possible if the heuristic helps.

Why not just fine-tune on verified synthetic chains, or retrieve documents up front? AbstRaL found that supervised fine-tuning alone taught the surface form of abstract reasoning without faithfulness, and RL was needed. Micro-Macro Retrieval shows that evidence retrieved at the start of the prompt fades over long reasoning chains. These results suggest neither approach removes the need for in-loop factual rewards. They motivate our design but do not replace the experiments.

Proposed Experiments

Training uses HotpotQA and 2WikiMultiHopQA prompts, with Wikipedia as the evidence corpus, following FSPO. Evaluation covers HaluEval-QA as the primary factuality metric, MATH-500 to confirm that reasoning ability survives, and exact match on HotpotQA, 2WikiMultiHopQA, and MuSiQue. We also run Micro-Macro Retrieval's HotpotQA-2Q and HotpotQA-3Q stress tests to see whether hop-level rewards resist the long-chain decay that paper documents.

Three checks decide the proposal. The first is whether our method at $q=0.1$ to $0.25$ beats budget-matched subsampled FSPO on HaluEval-QA by a margin larger than the spread across three random seeds. The second is whether it stays within two points of full FSPO's 83.0% on the 7B base, at a measured cost of at most a quarter of FSPO's checker calls and a clear reduction in wall-clock time per step. The third is whether the predictor-only variant visibly exploits its reward while the audited variant does not, as shown by the predictor-audit gap over training.

A fourth, diagnostic check applies OTA's monotonicity analysis to the factuality value along fully audited chains, comparing hop-level against sentence-level predictors. If horizon contraction yields fewer order violations and lower estimator variance at the same $q$, the temporal-abstraction component is doing real work. If it does not, the result is still a verifier-budgeting method, but with its reason for working stated correctly. Finally, we repeat the best 7B configuration at 14B and report whether factuality-aware RL becomes feasible at that scale under a fixed GPU-hour budget that full FSPO exceeds.