When a language model is fine-tuned with reinforcement learning, its reward usually mixes several goals, such as being correct, being informative, and being pleasant to read. A mix of this kind lets the model pay for a little more helpfulness with a little less truth. Factual accuracy should instead work like a safety rule, a priority that no amount of style can buy off. Lexicographic optimization, which ranks candidates by the first criterion and uses later criteria only to break ties, captures that priority. Applied naively, it fails in two ways. Factuality scores from automatic verifiers are continuous and noisy, so exact ties almost never happen and the secondary goals get no training signal. And a strict "most factual wins" rule teaches the model that saying nothing is safest. We propose a group-based policy-gradient method that addresses both failures. For each prompt, the model samples a group of responses. Any response whose verified factual precision falls within a tolerance $\tau$ of the group's best is treated as tied for first place. Only among these tied responses do informativeness, and then general quality, decide the ranking. The tolerance $\tau$ is not hand-tuned. It is set to the resolution of the verifier, meaning the smallest factuality gap the verifier can order correctly with high probability against human labels.
The claim is that this "factuality first, within measurement error" ordering does three things at once. It removes the model's incentive to trade factual claims for helpfulness. It still pushes the model to say as much true content as it reliably can, rather than retreating to hedges. And it gives a principled answer to an open problem in lexicographic reinforcement learning, namely how to choose the tolerance. If the method works, it should reach a better factuality–informativeness trade-off on long-form and short-form factual benchmarks than weighted-sum rewards, constrained optimization, or existing heuristic factuality penalties. It should do so without reducing reasoning accuracy.
Where this sits
Lexicographic reinforcement learning is well developed in control settings. Lexicographic Multi-Objective Reinforcement Learning keeps the actions whose primary value is within $\tau$ of the best, then optimizes lower-priority objectives within that set, and proves convergence. It tests only gridworlds and driving simulators, and it names choosing $\tau$ automatically as future work. Multi-Objective MDPs with Conditional Lexicographic Reward Preferences and Urban Driving with Multi-Objective Deep Reinforcement Learning show why the tolerance matters: without it, strict priority produces pathological caution, such as a car that never moves. These papers supply our mechanism. Our contribution is to carry it into language-model fine-tuning, where the "action" is a whole sampled response and the tolerance can be derived from the verifier rather than tuned.
Two papers connect ordered objectives to language-model training, and they are our closest competitors. From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning (RLRR) re-ranks each sampled group lexicographically by binary rule-checked correctness, then length, then a ranking reward model. It reports 64.8% average reasoning accuracy against 54.3% for plain GRPO, the group-based policy-gradient method that normalizes rewards within each prompt's group of samples. Its bins are hard and its first criterion is binary, so it neither covers graded open-domain factuality nor allows slack; a correct answer one token longer simply loses. The Hallucination Dilemma: Factuality-Aware Reinforcement Learning for Large Reasoning Models (FSPO) verifies each reasoning sentence against Wikipedia and flips the sign of the advantage on sentences that contradict the evidence. It raises HaluEval-QA accuracy for Qwen2.5-7B from 48.0% to 83.0%. FSPO is a strong penalty, but it is not an ordering. It has no notion of "close enough," and it gives no reward for informativeness, so abstention is never explicitly ranked against fabrication.
The motivating evidence is that outcome-only RL inflates hallucination: R1-Distill-Qwen-7B scores 11.6% on HaluEval-QA according to the same FSPO paper. Scalarized rewards invite the policy to sacrifice a low-weighted component, as documented in A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving. Extreme weight ratios destabilize optimization, according to Prioritized Optimal Control. Conformal Linguistic Calibration shows that the factuality–specificity trade-off can be controlled with statistical calibration at inference time, through sampling, clustering, and rewriting. We borrow its idea of calibrating a threshold on held-out data, but we apply the calibration to the training signal rather than to each query at run time.
Core Claim
For a prompt $x$ with sampled group $G = \{y_1, \dots, y_n\}$, let $F(y)$ be verified factual precision, defined as the fraction of atomic claims in $y$ supported by retrieved evidence. An explicit abstention counts as vacuously precise, $F = 1$. Let $I(y)$ be informativeness, defined as the number of supported claims relevant to the question. Let $Q(y)$ be a general quality score. The admissible set is
$$A_\tau(G) = \{\, y \in G : F(y) \ge \max_{y' \in G} F(y') - \tau \,\}.$$
Every admissible response outranks every inadmissible one. Admissible responses are ordered by $I$ and then by $Q$. Inadmissible responses are ordered by $F$. The resulting ranks are mapped linearly to advantages, as in RLRR's pure relative reward.
Three properties follow from this ordering. First, a confident fabrication can never outrank an honest abstention unless its precision is within $\tau$ of the best response in the group. Second, among near-maximally factual responses, the one with more true content wins, so hedging is not a free strategy. Third, the priority is relative to each prompt's own group, so the method does not need to know in advance how factual the model can be on that prompt.
The tolerance comes from the verifier. On a calibration set of response pairs with human claim-level labels, we set $\tau$ to the smallest precision gap $d$ such that the verifier orders pairs with $|F_{\text{human}}(y) - F_{\text{human}}(y')| \ge d$ correctly with probability at least $1-\alpha$. The rationale is that below the verifier's resolution, a factuality ordering is mostly noise. Following that noise wastes the gradient that the secondary objectives could use.
Research Program
We build a claim-level verifier following FSPO's recipe: decompose the response into atomic claims, retrieve Wikipedia evidence, and classify each claim as entailed, neutral, or contradictory. We then measure the verifier's error against human labels on a few hundred long-form answers and derive $\tau$ at several values of $\alpha$. The ordering above is implemented inside GRPO, with correctness-aware clipping of negative advantages borrowed from RLRR.
We run two tracks. The knowledge track fine-tunes Qwen2.5-7B-Instruct on fact-seeking prompts: biography requests in the style of FActScore and long-form versions of Natural Questions. The reasoning track reproduces the FSPO setting on Qwen2.5-7B. There, $F$ is computed over reasoning sentences and final-answer correctness sits between factuality and informativeness in the order.
Throughout both tracks we log a diagnostic we call the priority inversion rate. It is the fraction of within-group pairs in which the response with higher advantage has verified precision lower by more than $\tau$. It measures directly whether a training scheme is letting helpfulness buy off truth.
Baselines
The naive reference points are the untuned instruct model and standard GRPO with a correctness-only or reward-model-only signal. The latter shows how much outcome-driven training degrades factuality, as FSPO documents.
The strongest scalar competitor is GRPO with a weighted-sum reward, $w_F F + w_I I + w_Q Q$. We sweep $w_F$ over two orders of magnitude, including very large values, and pick the best setting on a validation set. Under group normalization, a large $w_F$ approximates strict priority, which makes this sweep the most dangerous comparison. The same sweep budget, number of seeds, training prompts, verifier, and number of policy updates will be given to our method, which searches only over $\alpha$.
The strongest existing factuality method is FSPO. On the reasoning track, the target to beat is its reported 83.0% on HaluEval-QA for Qwen2.5-7B, while matching its GSM8K accuracy. FSPO reports GSM8K rising from 65.2% to 89.5%. On the knowledge track we reimplement FSPO's sign flip on claim-level verdicts with the same verifier.
We also run an RLRR-style variant that binarizes factuality at a threshold before hard lexicographic sorting, and a Lagrangian GRPO variant that maximizes $I$ and $Q$ subject to mean hallucination rate at most $\epsilon$, with $\epsilon$ swept. As inference-time references, we include DoLa, applied through the DoLa repository, and Conformal Linguistic Calibration. For the knowledge-track metrics on FActScore/CORE and SimpleQA, no reported result found on Lacuna for these baselines on our exact configuration, so all will be rerun.
Two ablations isolate the key ingredient. The first uses strict lexicographic ordering with $\tau = 0$, which we expect to collapse toward abstention and terse answers. The second uses a hand-picked fixed $\tau$ in place of the verifier-calibrated one. A third variant uses an absolute factuality threshold rather than one relative to each group's best, which separates "relative to the group" from "slack."
Beating the baselines means reaching a factuality–informativeness curve that dominates the weighted-sum and Lagrangian curves across settings. Concretely, that is higher supported claims per answer at equal or better precision on FActScore/CORE, and a higher correct rate at an equal or lower incorrect rate on SimpleQA. It also means a lower priority inversion rate, and no loss relative to FSPO on HaluEval-QA or GSM8K.
Why not just…?
Why not just use a weighted sum with a very large factuality weight? This is the real threat. Group normalization rescales rewards within each prompt's group, which may remove the numerical instability that Prioritized Optimal Control describes. A large weight then behaves close to strict priority. But strict priority is exactly the $\tau = 0$ failure mode: the secondary terms carry almost no signal, and the policy drifts toward saying less. Smaller weights reopen the trade that A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving warns about. The weight sweep will show whether any single weight matches our trade-off curve and inversion rate. If one does, the honest conclusion is that the contribution reduces to the calibrated tolerance. We would then report the method as a way to choose that weight from verifier noise, not as a new ordering.
Why not just constrain the hallucination rate with Lagrange multipliers? A constrained formulation needs a global threshold $\epsilon$. The achievable factuality varies enormously across prompts: obscure entities versus famous ones. A single threshold is therefore infeasible on hard prompts and slack on easy ones. Lexicographic Multi-Objective Reinforcement Learning reports that lexicographic methods matched or beat the constrained methods RCPO and AproPO and failed gracefully when goals were unreachable. Our group-relative admissible set adapts per prompt without a global threshold. The $\epsilon$ sweep, with results broken down by entity popularity, tests this directly.
Why not just use FSPO's sign flip, or RLRR's bins with factuality as the first bin? Both already help. FSPO's 83.0% on HaluEval-QA is a high bar. Neither method, however, rewards informativeness among factual answers, so neither distinguishes a careful full answer from an abstention. RLRR's binarization throws away graded precision. If FSPO matches us on FActScore informativeness as well as on HaluEval-QA, our ordering adds nothing beyond a penalty, and the comparison on the knowledge track is designed to reveal exactly that.
Why not just fix hallucinations at inference time with DoLa or conformal hedging? Inference-time methods leave the trained policy's incentives unchanged. Conformal hedging pays for sampling, clustering, and rewriting on every query. DoLa cannot supply knowledge the model lacks, as its own limitations note. We will report latency together with accuracy. We will also test whether combining our trained policy with DoLa gives additive gains, which would show the two methods address different failures.
Decisive Checks
The central experiment plots trade-off curves on FActScore/CORE and SimpleQA, with abstention allowed. The x-axis is precision, or the incorrect rate on SimpleQA; the y-axis is supported claims, or the correct rate on SimpleQA. We compare our method against the full weighted-sum and Lagrangian sweeps, with TruthfulQA and HaluEval-QA as secondary checks and GSM8K guarding against reasoning regression.
The idea is supported if three things hold. The calibrated-$\tau$ curve should dominate the best scalar curve. The $\tau = 0$ ablation should visibly collapse in informativeness. And the priority inversion rate should be near zero for our method but clearly positive for weighted sums that reach comparable informativeness.
The idea is refuted if a tuned weighted sum reaches the same curve and inversion rate, or if a hand-picked $\tau$ does as well as the calibrated one. In the second case, the verifier-resolution argument for choosing $\tau$ would be unnecessary.
References
- Lexicographic Multi-Objective Reinforcement Learning
- Multi-Objective MDPs with Conditional Lexicographic Reward Preferences
- Urban Driving with Multi-Objective Deep Reinforcement Learning
- From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
- The Hallucination Dilemma: Factuality-Aware Reinforcement Learning for Large Reasoning Models
- A Review of Reward Functions for Reinforcement Learning in the context of Autonomous Driving
- Prioritized optimal control
- Conformal Linguistic Calibration: Trading-off between Factuality and Specificity