Order-Aware Claim Verification for Book-Length Documents

Models that check whether a claim about a long document is true tend to trust what they read first. When the evidence for a claim sits in chapter twenty rather than chapter two, or when the claim hinges on which of two events came first, the model often returns a confident verdict that the document does not support. We propose to measure and repair this failure with two ingredients. The first is synthetic training data, written by a language model, in which we control where the evidence sits and flip the stated order of events. The second is a verifier that must point to where each event in a claim happens in the document, and then pass a check that those locations respect the order the claim asserts.

The central idea fits in two sentences. Lead bias (over-reliance on early text) and order errors are the same underlying failure: the verifier does not track where in the document's timeline its evidence comes from. If we generate claims whose evidence location and event order are known by construction, we can supervise that location directly and turn "is the stated order consistent with the located evidence?" into an explicit, differentiable check.

Core Claim

The claim is that position-controlled, order-inverted synthetic claims, used to train a verifier that outputs a monotone alignment from claim events to document segments, will do two things. They will shrink the accuracy gap between claims with early and late evidence. They will also sharply reduce wrong verdicts on claims about event order. Neither result should come from simply adding more synthetic data or from position-agnostic chunk-level checking.

Three lines of evidence motivate this. Position bias in text models is large and measurable. An Empirical Study of Position Bias in Modern Information Retrieval builds position-bucketed question sets (SQUAD-POSQ, FINEWEB-POSQ) and finds that dense and late-interaction retrievers lose 15.6% on average when evidence moves later. The same study finds that full cross-encoders stay nearly flat, with a Position Sensitivity Index below 0.03. Collapse of Dense Retrievers shows that retrievers prefer foil documents with early keyword matches more than 90% of the time. It also shows that feeding those foils to GPT-4o drops accuracy by 34% relative to giving no context, so positional bias directly produces factual errors downstream. Both studies stop at retrieval. Neither asks whether the verifying or generating model itself is biased toward early positions, and neither looks at claims about order.

The method comes from video. In video, temporal sequence alignment means matching the steps of a procedure to the moments they occur while forbidding matches that go backward in time. SVIP reports that a sequence alignment loss gains over 6% AUC on its "alter-order" split. Video Action Differencing reports that replacing order-constrained frame alignment with random frames collapses accuracy to 50.1%. ViterbiPlanNet shows that such order-constrained search can be made differentiable by replacing max with log-sum-exp. We borrow that mechanism and move it from video frames to document segments.

The data comes from LLM-based augmentation. CLIPPER compresses books into chapter outlines before generating claims. This cuts the claim error rate from 73.1% to 16.7%, and fine-tuning Llama-3.1-8B-Instruct on its data raises long-form claim verification accuracy from 27.9% to 76.0%. CoCa-CXR shows that programmatically swapping temporal order and flipping the label ("improved" becomes "worsened") yields balanced contrastive pairs that teach progression. We combine the two. CLIPPER-style generation from outlines gives each claim a known chapter provenance for free, and CoCa-CXR-style inversion gives each order-bearing claim a matched false twin.

Research Program

The first step is diagnostic, and the rest of the project depends on it. We take the CLIPPER-trained verifier and several off-the-shelf long-context models. We bucket held-out claims by the relative depth of their evidence chapters and report accuracy per bucket, together with a position-sensitivity gap in the spirit of PSI. The gap is defined as the relative difference between the best and worst depth bucket. We separately report accuracy on claims whose truth depends on event order. If the verifiers show neither a depth gap nor an order deficit, the lead-bias half of this proposal is moot. In that case we would narrow to order verification alone.

The second step builds the data. From each book's CLIPPER-style outline, a generator LLM writes claims that mention two to four events. Each event is tagged with the chapter it came from. Three transformations are then applied. Depth rebalancing samples claims so that evidence depth is uniform. Order inversion rewrites temporal connectives ("after X, Y" becomes "before X, Y") or swaps the event order in the claim, and labels the result false. It is applied only when the outline confirms the two events occur in distinct chapters in a known order. Location-preserving distractors keep the events but attribute one of them to a similar event from a different chapter. Because provenance is known by construction, every claim comes with gold event-to-chapter alignments.

The generator itself can hallucinate provenance. We therefore follow the lesson of CLIPPER and generate only from compressed outlines. We also audit a sample by hand. Following the approach of ProcessBench, which discarded 30% of its initial data for lack of annotator agreement, we expect to discard ambiguous items rather than force labels on them.

The third step builds the verifier. A Llama-3.1-8B-Instruct verifier, matching CLIPPER's model for comparability, reads the book with its chapter outline. For each event $i$ in the claim, it emits a distribution $p_{ij}$ over chapters $j$. Given the order the claim asserts, a soft dynamic program scores two quantities. $A_{\text{mono}}$ is the log-sum-exp over all chapter assignments that respect the claimed order, computed with the same smoothing ViterbiPlanNet uses. $A_{\text{free}} = \sum_i \log \sum_j p_{ij}$ is the unconstrained score, which ignores order. The order-violation signal is

$$\Delta = A_{\text{free}} - A_{\text{mono}},$$

which is near zero when the most likely locations already satisfy the claimed order and grows when they contradict it. The final verdict combines the model's own true/false logit with $\Delta$. Training uses verdict cross-entropy plus a cross-entropy loss on $p_{ij}$ against the gold alignments from the synthetic data. The alignment loss is what forces the model to find late evidence rather than settle for early lookalikes.

Three limits are known in advance. Many claims involve a single event or concurrent events. For those, $\Delta$ is zero by construction and the model relies on its verdict logit alone. This mirrors the known limitation of alignment methods to strictly sequential steps. Stories told out of chronological order, such as flashbacks, break the assumption that chapter order equals story order. We will measure this failure rather than hide it. Finally, CLIPPER itself notes that synthetic claims miss the situational nuance of human benchmarks like NoCha, so gains on synthetic test sets need not transfer.

Baselines

The naive reference point is the base Llama-3.1-8B-Instruct model prompted zero-shot for true/false with the full book in context. CLIPPER reports 27.9% for this setup on its claim verification test set. It anchors how much any training helps and gives a first read on raw lead bias.

The strongest existing method is CLIPPER fine-tuning itself, reported at 76.0% accuracy for Llama-3.1-8B-Instruct in CLIPPER. It is the closest competitor because it trains on the same kind of outline-grounded synthetic claims with chain-of-thought rationales. Any improvement we show could otherwise just be more or different synthetic data. We will rerun it on our splits with the same base model, the same number of training claims, and the same number of tuning trials over learning rate and epochs as our method. Beating it means two things. Overall accuracy must be at least equal. The worst-depth bucket and the order-inverted pairs must improve by a margin larger than the seed-to-seed spread across three runs.

A position-robust pipeline baseline tests whether a chunk-level check already solves the problem. It splits the book into chapters and scores each chapter against the claim with a full cross-encoder reranker. The IR study found bge-reranker-v2-m3 to have PSI below 0.03 and NDCG@10 between 93.53 and 96.10 on its position benchmarks. The top chapters are then passed to the same 8B verifier. It receives the same training data and budget. On single-event claims, it may match us.

Four ablations isolate our ingredients. The first is CLIPPER data with depth rebalancing only. The second adds order inversion but no alignment head, trained with verdict loss only. The third uses the alignment head without the monotone constraint, so the verdict uses $p_{ij}$ but not $\Delta$. The fourth removes the gold alignment supervision. Order inversion without the alignment head is the key ablation. If it matches the full model, the dynamic program is unnecessary, and the contribution reduces to a data recipe.

Why not just…?

Why not just shuffle or rotate chapters as augmentation, with no LLM involved? Moving chapters around does spread evidence across depths, so it could plausibly fix lead bias alone. That makes it a real threat on the depth metric. But it destroys the document's timeline, which turns every order-bearing claim into noise. We will train a chapter-rotation variant with the same budget. We expect it to close much of the depth gap while matching or hurting accuracy on order-inverted pairs. If it closes both, our order machinery is not needed, and we will say so.

Why not just verify claim-chapter pairs with a cross-encoder, since rerankers are known to be position-robust? The IR study's reranker result is exactly why this is our pipeline baseline and not a strawman. The limitation is structural. A per-chunk score cannot tell whether event A's chapter precedes event B's unless something downstream compares locations. Adding that comparison is what our alignment step does. The experiment on order-inverted pairs, where both chapters are retrieved correctly but the stated order is wrong, settles whether the downstream verifier already handles this implicitly.

Why not just train longer on more CLIPPER data, or use a bidirectional or adversarial verifier? The Bidirectional Process Reward Model reports a 37.7% step-level F1 gain on ProcessBench by adding a right-to-left pass. That gain shows reading direction matters for catching errors whose problems appear only later. Adversarial Training for Process Reward Models reports gains of 3.4 points overall and 5.3 points out of distribution from generated hard negatives. Both work on reasoning steps, not document positions, and neither supervises where evidence lies. A scaled-data CLIPPER run at twice our training set size will test whether volume alone closes the order and depth gaps. A reversed-order-reading variant will test whether a BiPRM-style second pass does the same.

Proposed Experiments

The main evaluation uses CLIPPER's held-out book claims, re-bucketed by evidence depth into five equal bins. We add a new order test set. It consists of true order-bearing claims paired with their inverted twins, and it is hand-verified on a sample so we are not grading only against the generator. The metrics are overall accuracy, worst-bucket accuracy, the position-sensitivity gap, and paired order accuracy. Paired order accuracy counts a pair as correct only if both the claim and its inverted twin are judged correctly, which blocks the shortcut of always answering "true." We also measure rationale grounding, the fraction of chapters the verifier points to that actually contain the event. It serves as a direct hallucination measure on the explanation, not just the verdict.

For transfer, we evaluate on NoCha, for which no reported result was found on Lacuna for any of these baselines. The file only notes that CLIPPER's claims miss its nuances. To test whether lead bias is document-type specific, we run the diagnostic on BIGPATENT-derived claims. BIGPATENT shows that patents spread salient content uniformly and that LEAD-3 fails there.

Three decisive checks follow. First, the full model must beat order inversion without the alignment head on paired order accuracy. If it does not, the video-derived alignment adds nothing in text. Second, the depth gap must shrink relative to CLIPPER without a loss in overall accuracy. Third, performance on flashback-heavy books, which we will flag by manually listing out-of-chronology chapters in a small subset, will show whether the monotone assumption breaks where we expect. Together these tell us whether where the evidence sits and when events happen are one problem worth modeling jointly, or two problems that cheaper fixes already handle.