One-Claim Preference Pairs: Teaching LLMs Factuality from a Few Hundred Prompts

Large language models state false facts fluently, and the usual fix during training is preference tuning. The model is shown two whole responses, told which one is better, and nudged toward it. When the goal is factual accuracy, this signal is weak. Two long responses differ in style, length, and a dozen claims at once, so the model receives a vague signal about which difference mattered. Work on hallucination detection has shown that facts are best checked one at a time, as small "subject, relation, object" statements called claim-triplets. That work has so far been used only to grade models after the fact, never to decide what the training data should look like.

We propose to build preference pairs that differ in exactly one claim. We take a model's own response and split it into claim-triplets. Each triplet is checked against a knowledge graph, a database of verified entity relationships. We then make surgical edits. In one kind of edit, a correct claim is swapped for a plausible wrong one drawn from the graph's structure, for example the correct birthplace replaced by another city that often appears as a birthplace for similar people; this yields a hard rejected example. In another, an unsupported claim is corrected or replaced with an explicit hedge; this yields a chosen example. Because each pair isolates a single fact, every training example carries a concentrated factual signal. Our central claim is that this makes factuality alignment far more sample-efficient than standard response-level preference data. Specifically, one-claim pairs built from a few hundred prompts should cut closed-book hallucination about as much as response-level pairs built from ten times as many prompts.

Where this sits in prior work

Three detection systems motivate the unit of supervision. Knowledge-Centric Hallucination Detection (REFCHECKER) splits responses into claim-triplets and labels each against references as entailed, contradicted, or neutral. It reports that triplet-level checking beats sentence-level checking by about 10 Macro-F1 points. FactAlign aligns triplets from source and output in embedding space to classify distorted facts versus added facts. GCA builds graphs over triplets drawn from multiple sampled responses. It reaches 92.7 F1 on TripleCom, a benchmark of real entities joined by fabricated relationships. All three stop at detection. None of them feeds its verdicts back into training the generator, and that loop is what we close.

MultiRAG also uses triplet graphs against hallucination, but only at inference time, as a confidence filter over retrieved sources. Its authors list parameter-efficient alignment as unexplored. Our method changes the model's weights and needs no graph at test time.

Two papers supply our method. BPO shows that preference data can be made roughly 10× more efficient by choosing prompts carefully. On SafeRLHF with Llama-3-8B, it reports an MT-Bench score of 5.45 from 4,570 samples, versus 4.90 for vanilla direct preference optimization (DPO) on more than 50,000 samples. BPO targets conversational quality with GPT-4 scoring and was never evaluated on factual benchmarks. We ask whether the efficiency can come from the structure of each pair rather than from prompt selection. The multimodal study Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data supplies the corruption idea. Replacing grounded phrases with plausible fakes raised F1 with 500 labels from 17.98% to 25.30%, and corrupting grounded spans worked far better than corrupting random spans. We carry this finding from image grounding to knowledge-graph grounding, and from training detectors to training generators.

Two papers constrain the design. Hallucination is a Consequence of Space-Optimality proves that a model with finite capacity cannot memorize long-tail facts without some hallucination, and that confidence thresholds alone cannot fix this. We therefore do not expect the model to learn every fact. Instead, the "neutral" triplets, those the graph cannot confirm, become hedge-versus-assert pairs that teach the model when not to commit. REFCHECKER's finding of "internal knowledge bias", where model-based checkers fall back on their own memory, is a warning about circularity. GraphCare and KG-FIT both let GPT-4 write the very knowledge they rely on. To avoid that, we check claims against curated graphs rather than against an LLM's opinion. We also measure directly how errors in the graph propagate into training.

Research Program

The pipeline has four steps.

First, a base model of about 8B parameters, the Llama-3-8B scale that BPO used, answers entity-centric prompts closed-book. These are biographies and entity descriptions over entities in YAGO3-10 and FB15K-237, and biomedical entity questions over PrimeKG.

Second, an open triplet extractor splits each response into claims. Each claim is labeled entailed, contradicted, or unverifiable, by matching against the graph with a natural language inference check. No LLM judge is used in this step.

Third, we build pairs by single-claim edits. An entailed claim yields a rejected variant: the object is swapped for a graph "sibling", an entity with the same relation type and type constraints. A contradicted claim yields a chosen variant: the object is set to the graph's value. An unverifiable claim yields a pair that prefers a hedged version over the confident one. The surrounding text is held fixed, so the only difference between the two responses is one factual span.

Fourth, we train with standard DPO, keeping the prompt budget matched across all methods.

The analysis has three parts. First, we measure hallucination rate as a function of the number of training prompts, at 250, 500, 1k, 2k, and 5k prompts, to estimate how much data the method saves. Second, we compare sibling swaps with random swaps, to test whether the graph's structure is what supplies the value. Third, we deliberately corrupt 5%, 15%, and 30% of the graph's edges, to measure how noisy knowledge sources degrade the method. This speaks to the setting where GraphCare had to rely on expert verification.

Baselines

The naive reference point is the untuned base model, with and without an instruction to avoid uncertain claims. This shows how much of any gain is just prompting.

The second baseline is supervised fine-tuning on verified responses: the base model's own outputs with contradicted and unverifiable claims deleted. It uses the same prompts and the same graph. If it matches our method, the preference contrast adds nothing.

The strongest existing competitor is response-level DPO with prompts chosen as in BPO, with its K-means breadth compression. Chosen and rejected responses are drawn from multiple samples and ranked by an LLM judge. This is the best published recipe for data-efficient preference alignment. It will receive the same prompt budget, the same number of pairs, and the same learning-rate and β sweep. Beating it means a lower rate of contradicted triplets on KNOWHALBENCH's zero-context split at equal prompt count, with no loss on MT-Bench. On factual benchmarks there is no reported result found on Lacuna for BPO or for any triplet-driven DPO. The factual reference numbers in the file are from TruthfulQA: the best GPT-3 result was 58% truthful against 94% for humans. The file also cites a survey report, Siren's Song, that RLHF raised TruthfulQA accuracy from about 30% to 60%.

The key ablation is response-level DPO scored by the same triplet checker. Pairs are ranked by the count of contradicted claims, so the knowledge signal is identical but the pairs are not minimal. This separates "triplet-level verification" from "one-claim pair construction". A second ablation replaces graph-sibling swaps with random-entity swaps.

Why not just…?

Why not just rank whole responses by the triplet checker and run ordinary DPO? This is the most serious threat, and it is our main ablation. The case against it is that whole responses differing in many claims give a diluted gradient. The multimodal corruption result suggests that targeted, grounded contrasts carry more signal per example. If checker-ranked response pairs match our method at every data budget, we will report that the checker, not the pair construction, is the contribution.

Why not just retrieve at inference time? Retrieval helps when a clean source exists. But KNOWHALBENCH's noisy-context setting and the medical summarization study, where fuller context increased errors, show that retrieval does not by itself stop hallucination. Retrieval also leaves closed-book behaviour unchanged. We will evaluate both the zero-context and noisy-context splits, and a combination of tuned model plus retrieval. Retrieval may well win on accuracy in the retrieved setting, so our claim is limited to parametric behaviour and to whether tuning makes the model more robust under retrieval.

Why not just filter with best-of-n sampling and a checker? This works, but it multiplies inference cost by n and needs the graph at deployment. GCA's multi-sample pipeline shows how costly that can become. We will report best-of-8 with REFCHECKER as an inference-time upper reference.

Why not just use a bigger model? TruthfulQA shows that larger models can be less truthful. The space-optimality theorem says long-tail facts cost exponentially more memory. We include a larger untuned model as a reference point, but scale is not a like-for-like alternative.

Decisive Checks

Evaluation uses four benchmarks. The first is KNOWHALBENCH's zero-context and noisy-context splits, scored with REFCHECKER against held-out references. The second is HaluEval QA and dialogue. The third is TruthfulQA, scored with GPT-judge. The fourth is long-form biography factuality on WikiBio and PHD entities excluded from training, scored by the rate of contradicted triplets. Because training and scoring both use triplets, we add a human audit of 300 sampled claims to check that the model has not learned to fool the checker.

The method succeeds if three things hold. At 500 prompts, it must match or beat response-level BPO-style DPO at 5,000 prompts on contradicted-claim rate. It must beat the checker-ranked response-DPO ablation at the same budget. And its hedging rate on graph-verified facts must stay near the base model's, so the gains do not come from refusing to answer.

The method fails if any of three things happen. The gap to the ablation may vanish above 1k prompts, echoing SciPrompt's shrinking advantage as labeled data grows. Held-out entities may show no gain, meaning the model memorized the edited facts instead of learning when to commit. Or 15% graph corruption may erase the benefit, meaning the approach cannot tolerate realistic knowledge sources.