Language models often state false facts in long answers that they would get right if asked directly. A biography generator might invent a birth year or an award, even though the same model "knows" the person well enough to pick them out from a description. This note proposes a cheap, training-free test that runs each generated claim backward. Instead of asking "is this claim true of person X?", we ask "does adding this claim make it easier or harder for the model to work out that the description is about X?" A true claim should point toward the right person. A fabricated or swapped claim should point nowhere, or toward someone else. The claim is that this backward signal makes different mistakes from forward self-checks, which ask the model to re-answer the claim in the same direction it was produced. If so, combining the two should catch hallucinations that either misses alone.
The idea draws on four lines of work, unevenly. Reverse generation, meaning anchoring on the end of a sequence and reasoning toward the beginning, supplies the core mechanism. Hallucination benchmarks supply the application. In-context learning, where the model is taught a task through a few worked examples in the prompt rather than through training, is how the backward queries are built. Hyperparameter optimization is a constraint on how all methods are compared: every method gets the same tuning budget. That constraint is what makes the comparison credible, because published verification pipelines are run at fixed default settings.
Core Claim
Two published results seem to contradict each other. The Factorization Curse shows that left-to-right models trained on "A is B" score 0% on "what is B?". It attributes this to next-token training learning only one direction. Yet Reverse Validation (RV) asks a frozen model to name the entity a generated passage describes, and detects passage-level hallucinations at 59.1 F1 on its PHD benchmark. FOBAR masks a number in a math problem and solves backward for it, improving GSM8K accuracy from 67.9% to 70.8% with text-davinci-003.
Our reading is that backward checks work when the backward query is many-to-one and well represented in pretraining text. Identifying a well-known person from a bundle of their attributes is such a query. Backward checks fail when only one direction was ever seen. If this is right, two things follow. Backward verification should be most useful exactly where forward verification is least reliable. And its value should collapse on tail entities, which gives a testable boundary rather than a hope.
The concrete new piece is a backward score for each atomic claim. RV works on whole passages and, by its own account, breaks on passages with several interlinked claims. FOBAR only works on invertible numerical problems. Our score works claim by claim on free text.
Let $e$ be the subject entity, $S$ the set of atomic claims extracted from a generation, and $c \in S$ one claim. The backward evidence of $c$ is how much including $c$ changes the model's log-probability of naming $e$ when shown the other claims:
$$B(c) = \log P_\theta(e \mid S) - \log P_\theta(e \mid S \setminus \{c\})$$
Here $P_\theta$ is the verifier model's probability of the entity name. It is computed by forced decoding after an in-context prompt that teaches "these facts describe: ___". We also compute $B$ against a few confusable entities that the model itself proposes, so that a swapped claim can be caught when it raises the probability of the wrong person. A forward score $F(c)$ comes from factored short-form verification in the style of Chain-of-Verification (CoVe), which answers each check in a fresh context. The final decision combines the two, generalizing FOBAR's geometric mean to a tuned weight $\alpha$:
$$\text{score}(c) = \alpha\, F(c) + (1-\alpha)\, \hat{B}(c)$$
In this formula, $\hat{B}$ is $B$ standardized within each generation.
Research Program
The work has three parts.
Building the scorer. First we build the scorer on an open-weight model, because the scorer needs token log-probabilities. Atomic claims are extracted following the FActScore procedure. Backward prompts use a small fixed set of in-context demonstrations. The verifier can be a different model from the generator, so it can also score the ChatGPT-written passages in PHD.
Testing the mechanism. Second, we test the mechanism, not just the end score. The central measurement is error correlation: across labeled atomic claims, how often do forward and backward checks fail on the same claim? If backward checks merely repeat forward retrieval, the correlation will be high and the combination will add little. We stratify every result by entity popularity, using PHD's Low/Medium/High search-volume strata and FActScore's rarity buckets. Hallucination prevalence in PHD is 40% for Low versus 14% for High, so the tail is where the stakes are.
Matched tuning. Third, we tune every method under the same protocol. Decoding Decoded shows that decoding settings can matter more than model scale for generation quality. Hyperparameter sensitivity work on character transduction shows that untuned baselines can flip architectural conclusions. CoVe, RV and FOBAR all report fixed settings. We give each method the same number of random-search trials on a development split. The search covers temperature, number of samples, prompt variant and decision threshold, and also $\alpha$ for our method. Sensitivity is reported with that paper's relative-performance-at-$k$ metric, so a reader can see whether a gain survives the tuning budget.
Baselines
Do nothing. The naive reference is the generator's raw output, scored by FActScore. For detection, the reference is a forward token-probability threshold on each claim. Any method must beat these at equal claims retained. Otherwise it is only deleting text.
CoVe factor+revise. This is the strongest existing competitor for mitigation. It is closed-book and needs no retrieval. Its factored variant already avoids the problem of the model copying its own draft. Its reported result is a FActScore improvement from 55.9 to 71.4 on Llama 65B biographies. We re-run it ourselves on the same open model, the same biography entities, and the same tuning budget. Beating it means higher FActScore at a matched number of retained facts per biography, at equal or fewer forward passes. We will also report a version where CoVe gets our extra compute as additional verification samples.
RV-EM. For passage-level detection, the strongest reported number is RV-EM's 59.1 F1 on PHD, against 41.1 for SelfCheckGPT and 25.5 for LMvsLM. We re-run RV-EM with our verifier model and tuning budget. Beating it means higher F1 overall and, more importantly, on PHD-Low, where RV's single-entity matching is weakest.
Ablations. Three ablations isolate the key ingredient. The forward-only ablation ($\alpha=1$) and the backward-only ablation ($\alpha=0$) show whether the combination, rather than either half, drives any gain. A shuffled-entity control computes $B$ against a random unrelated entity. If that control "works", the backward score is just measuring claim fluency.
Out of scope. For WikiBio, Generation Constraint Scaling reports RougeL 0.72 with Larimar. That number relies on an external memory architecture, so we cite it only as context and not as a matched baseline.
Why not just…?
Why not just run CoVe with more forward samples? This is the real threat. If backward errors are as correlated with forward errors as two forward samples are with each other, then spending the same compute on more CoVe samples will match us. FOBAR is suggestive here. In roughly 60% of its self-consistency failures, the right answer appeared in a minority chain but was outvoted, and only the backward check recovered it. That was math, though. The compute-matched CoVe run and the error-correlation measurement settle this directly. We will report a negative result if the correlation is no lower than between forward samples.
Why not just use RV at the passage level? RV gives one verdict per passage. That is useless for repairing a biography that is 80% correct. RV's own analysis also reports failures on multi-entity, multi-claim text. The leave-one-out form of $B(c)$ is designed to attribute blame to individual claims. Atomic-level FActScore labels test whether it does.
Why not just retrieve from the web? Retrieval is a stronger fix when it is available, and we do not claim otherwise. The case for a zero-resource check rests on settings without search, and on niche entities that are poorly indexed. CoVe's closed-book Llama 65B also beat PerplexityAI on frequent facts. We restrict claims to the closed-book setting.
Why not just prompt the model to generate the ending first? Terminal anchoring works in poetry (Automatic Poetry Generation from Prosaic Text). But a factual biography has no single terminal constraint to anchor on. Moreover, the Factorization Curse suggests that prompting a forward model to produce reversed text does not give it reverse knowledge. Using reverse reasoning as a verifier sidesteps that limit.
Decisive Checks
Three outcomes decide the proposal.
Error correlation. The phi coefficient between forward and backward failures on labeled FActScore claims must be clearly below the correlation between two independent forward verification samples. If it is not, the idea is wrong.
Matched-compute mitigation. At matched compute and tuning budget, the combined score must raise FActScore above re-run CoVe factor+revise at equal facts retained. It must also beat RV-EM's F1 on PHD-Low.
The popularity boundary. The popularity stratification must show the predicted shape. Backward evidence should add the most for mid-frequency entities and fall toward zero for the rarest. That would locate where the reversal curse begins to apply to frozen models. If backward gains are flat across popularity, our explanation of the RV–Factorization Curse contradiction is wrong, even if the method still helps.
References
- The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
- A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection
- Forward-Backward Reasoning in Large Language Models for Mathematical Verification
- Chain-of-Verification Reduces Hallucination in Large Language Models
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation
- Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
- Generation constraint scaling can mitigate hallucination
- Automatic Poetry Generation from Prosaic Text