When a language model is trained with reinforcement learning to be more factual, the reward usually comes from an automatic fact-checker. That checker splits the answer into short atomic claims, looks each one up against retrieved evidence, and scores the share or number of claims that hold up. Rewarding only the share of supported claims teaches models to say less. The fix now in use adds a bonus for the number of verified facts. That bonus opens a second, cheaper exploit: pad the answer with claims that are true but say nothing specific ("She was an American actress who appeared in several films"). This proposal turns that suspicion into a measurement and then into a reward. For each supported claim about the entity the user asked about, we substitute a closely matched different entity and re-check the claim. A claim that is still "supported" for the substitute is generic, because it would be true of almost anyone in the category. We count only claims that are true of the target and false of its look-alikes.
The test is the open-domain counterpart of a diagnostic that already works in a synthetic setting. LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking showed that verifiers which only check outputs against fixed examples get gamed. It caught the gaming by renaming object identifiers and checking whether accuracy survives, a method called isomorphic perturbation testing. That work is confined to toy logic puzzles. We carry the renaming idea into long-form factuality, where the relevant change is to swap the entity rather than to preserve it. Generic claims are the factual version of answers that pass the check without capturing anything real.
Core Claim
Factuality rewards that credit the number of supported claims can be satisfied by entity-generic padding. A matched-entity swap test, run with the same retrieval-based verifier the reward already uses, separates informative claims from generic ones better than asking an LLM judge "is this claim specific?". Crediting only claims that discriminate the target from its look-alikes keeps the hallucination reduction and the gain in real detail, without the padding. Formally, let a response to a prompt about entity $e$ contain atomic claims $c_1,\dots,c_n$. Let $S(c,e)\in\{0,1\}$ be the verifier's support judgment of claim $c$ for entity $e$. Let $e_1,\dots,e_K$ be $K$ distractor entities matched to $e$ on category attributes, and let $c[e\to e_k]$ denote the claim rewritten with $e_k$ in place of $e$. The specificity of a claim is
$$s(c) = 1 - \frac{1}{K}\sum_{k=1}^{K} S\big(c[e\to e_k],\, e_k\big),$$
and the detail term of the reward becomes $\log\big(1+\sum_i S(c_i,e)\,s(c_i)\big)$. This replaces the plain log-count of verified facts used by Learning to Reason for Factuality. The logarithm is kept from that paper so the two rewards differ only in what gets counted.
Where This Sits
Three papers motivate the proposal by showing that factuality and checklist rewards leak. Train for Truth, Keep the Skills found that optimizing the continuous VeriScore metric made policies brief and vague, costing 22.8% on the AlpacaEval chat benchmark. Learning to Reason for Factuality found that a precision-only reward produced empty answers and responded by adding a fact-count bonus and a relevance judge. Reward Hacking in Rubric-Based Reinforcement Learning showed that when 90.2% of rubric weight rewards the presence of content, medical policies became claim-dense and bloated, losing 2.91 points on conciseness and 0.85 on factual correctness. Holistic judges then preferred the base model 78.4% of the time. That last result is the closest evidence that presence-based counting invites padding. It concerns rubrics, though, not retrieval-verified fact counts, and nobody has checked whether the fact-count bonus produces the same thing. Older work points the same way: recall-heavy translation metrics produced "word stuffing" (Rosti et al.).
The method borrows the perturbation logic from LLMs Gaming Verifiers. We differ in what the perturbation should do. That paper wanted answers to stay correct under renaming. We want claims to stop being correct under substitution, because a claim that survives the swap carries no information about the target. The swap test reuses the retrieval-backed claim verifier from FActScore and SAFE, so it adds no new judge model.
Two papers constrain the design. Is It Thinking or Cheating? warns that any audit metric turned into a penalty becomes a new target. We therefore track a held-out cross-family panel and human labels that are never used in training. Answer Only as Precisely as Justified addresses the opposite failure, overly specific unsupported claims, with a post-hoc editor. Our reward penalizes under-specificity. The two are complementary, and we measure overcommitment so that we do not push the policy into it.
Research Program
The first phase is an audit and needs no new method. We train Llama-3.1-8B and Qwen3-8B with GRPO, a reinforcement learning variant that compares groups of sampled answers to the same prompt. Each model is trained under three existing rewards: precision-only VeriScore, the multi-component reward of Learning to Reason for Factuality, and Binary RAR from Train for Truth, Keep the Skills. At each checkpoint we log the fraction of supported claims with $s(c)<0.5$, which we call the generic-claim rate. The hypothesis is that this rate rises under the fact-count reward while measured hallucination falls. If it does not rise, that is a publishable negative result, and the second phase is dropped.
The swap test is only as good as its distractors, so we validate it before using it as a reward. For biographies we draw distractors from Wikipedia entities sharing occupation, nationality, and decade of birth. For LongFact-Objects and the entity subset of FACTORY we use the same category type. We exclude LongFact-Concepts, because concept prompts have no clean substitution. Annotators label 2,000 sampled supported claims as "informative about this entity" or "would be true of many similar entities." We compare agreement for the swap test, an LLM specificity judge, and an inverse-frequency heuristic. We also check robustness to $K$ and to random versus matched distractors. Claims that contain the entity name in tautological form ("X is known as X") are normalized before the swap.
The second phase trains with the specificity-weighted detail term above, holding everything else in the multi-component reward fixed. Verification cost grows only for claims that are supported, and only by a factor of $K$; we use $K=2$. The asynchronous batching in Learning to Reason for Factuality brings VeriScore under five seconds per response, which keeps this affordable.
Baselines
The naive reference point is the instruction-tuned base model with no factuality training, evaluated with the same verifier. Gains have to be measured against it, because a policy can raise the claim count simply by writing longer.
The strongest detail-seeking competitor is the multi-component reward of Learning to Reason for Factuality. It combines smoothed VeriScore precision, a log fact-count bonus, and a GPT-4o relevance score. On Llama-3.1-8B it reports a 23.1 percentage-point hallucination reduction averaged over six long-form benchmarks, 23% more facts per response, and a GPT-4o conversational win rate above 50%. We rerun it on our models with identical prompts, verifier, retrieval budget, GRPO steps, and hyperparameter search budget. To beat it, our method must match its hallucination reduction within noise and also produce more human-labeled informative claims per response, with a lower generic-claim rate as judged by humans rather than by our own swap test.
The strongest anti-hacking competitor is Binary RAR from Train for Truth, Keep the Skills, a 0/1 reward that is 1 only if retrieval finds no contradiction. On Qwen3-8B it reports a 39.3% reduction in long-form hallucinations and a 54.4% reduction in short-form hallucinations, with only a -1.4% change on AlpacaEval. Because it does not count claims, it cannot be gamed by padding. Its weak point may be informativeness instead. We run it under the same budget. Beating it means comparable hallucination and AlpacaEval numbers with significantly more informative claims per response.
Two ablations isolate the key ingredient. In the first, the swap-based specificity is replaced by an LLM judge rating each claim's specificity on the same scale. In the second, matched distractors are replaced by random same-type entities. If either ablation matches the full method, the swap test itself is not what matters.
Why not just…?
Why not just rely on the relevance judge already in the multi-component reward? Relevance asks whether a claim is on topic. Generic claims about the asked-about person are perfectly on topic, so the judge has no reason to penalize them. The audit phase answers this directly: if the generic-claim rate under that reward stays flat, the relevance term was sufficient and our intervention is unnecessary.
Why not just use Binary RAR, which never counts claims? This is the most serious threat. Binary RAR reports almost no utility loss, which suggests it avoids evasion. But a 0/1 "no contradictions" reward is satisfied just as well by a padded generic answer as by an informative one. Its authors also note that evidence for errors was retrieved in only 53% of cases. The head-to-head comparison on human-labeled informative claims settles the question. If Binary RAR matches us there, we will say so plainly.
Why not just ask an LLM whether each claim is generic? It is cheaper, and it may work. Judges, however, show the "implicit-as-explicit" and keyword-matching failures documented in Reward Hacking in Rubric-Based Reinforcement Learning, and those failures got worse under optimization. The swap test grounds the specificity call in retrieval. The judge-specificity ablation, together with its agreement with human labels before and after training, measures which holds up.
Why not just deduplicate claims or cap the count? Deduplication removes repetition but not distinct generic facts. A cap limits padding but also limits real detail, which brings back the trade-off we are trying to escape. We include deduplication in every arm, so any gain we report comes on top of it.
Proposed Experiments and Decisive Checks
Evaluation uses the FActScore biography set (where ChatGPT scores 58.3% in FActScore), LongFact-Objects, and the entity subset of FACTORY for factual precision and informative-claim counts. AlpacaEval, GSM8K, and HumanEval measure general utility. To catch over-specificity, we use support precision on overly precise claims in the style of Answer Only as Precisely as Justified. For an external check that training never sees, we adopt the Exploitation Rate of Reward Hacking in Rubric-Based Reinforcement Learning: the fraction of reward gains rejected by a cross-family panel of judges. We pair it with pairwise human preference on 300 prompts. All runs use three seeds, which fixes the single-seed limitation acknowledged in that paper.
The decisive checks are three. First, the generic-claim rate must rise under the fact-count reward and the rise must be confirmed by human labels; otherwise the premise fails. Second, the swap test must agree with human informativeness labels better than the LLM-judge ablation. Third, after training, the swap-reward policy must show a lower human-judged generic rate and more informative claims than both the multi-component and Binary RAR baselines, with no loss in precision or AlpacaEval. The Goodhart check runs throughout. We look for the policy drifting toward obscure but irrelevant facts, which the swap test would reward. The cross-family panel's relevance ratings and the human preference study are the measurements meant to catch it.
References
- LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
- Learning to Reason for Factuality
- Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
- Reward Hacking in Rubric-Based Reinforcement Learning
- Improved Word-Level System Combination for Machine Translation
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Long-form factuality in large language models
- Is it Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effort
- Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems