Load-Bearing Citations for RL-Trained Document Readers

Models that answer questions about long, visually rich documents (fifty-page reports, slide decks, forms, papers with tables and charts) are increasingly trained with reinforcement learning (RL): the model is rewarded only when its final answer is right, and it learns on its own how to search pages and reason. In other settings, this kind of outcome-only training is known to make the model's written reasoning drift away from honest, readable language. Reasoning models emit gibberish that still carries real computation, and vision-language models trained with RL write explanations that grow less faithful to what actually drove their answers. Nobody has checked whether this drift happens when the model reads documents. That is exactly the setting where a reader most needs to trust why the model said a contract clause or a table cell supports its answer.

We propose to make each reasoning step in a document model point at a specific region of the document, and to reward these pointers only if two things hold. The quoted content must actually appear in the cited region, which is a cheap string check against the document's own text-recognition output. And the answer must depend on that region: blanking the cited regions should lower the model's confidence in its answer more than blanking random regions of the same size. The core bet is that documents, unlike natural images or charts, come with machine-checkable evidence (words with bounding boxes). That lets us build a grounding reward that resists the gaming that sinks judge-based faithfulness rewards, and anchors meaning the way image grounding did in early language-drift work, at the level of individual reasoning steps instead of whole outputs.

Core Claim

Outcome-rewarded RL for multi-page document question answering will, like RL elsewhere, make the model cite less and less of what it actually uses. A reward that requires citations to be both textually correct and causally necessary will keep accuracy close to that of outcome-only RL while sharply raising the fraction of correct answers whose stated evidence is real. Necessity is the essential half. A quote-matching reward alone should produce decorative citations: true quotes from regions the model does not rely on. This is the document analogue of the "token flipping" that Countering Language Drift via Visual Grounding found when it constrained only syntax.

Where This Sits

Three papers motivate the proposal by showing the drift. Reasoning Models Sometimes Output Illegible Chains of Thought finds that RL-trained reasoners produce illegible text that still does work: truncating it costs QwQ 53% of its relative accuracy. On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs shows faithfulness degrading over RL steps on natural-image and geometry tasks. Beyond Answer Correctness finds that 87.1% of a chart model's correct answers rest on hallucinated explanations. None of these studies documents, multi-page evidence, or layout.

Two papers constrain the design. Zhao et al. report that putting a faithfulness score directly into the RL reward can collapse training onto shortcuts. Dahiya and Saini's external verifier left 47% of claims "uncertain," which means judge-based rewards are noisy. We therefore avoid any language-model judge in the reward and use only deterministic string checks and the policy's own answer likelihood.

Three papers supply the method. ALDEN gives the training setting: an RL agent with search and fetch page actions, with KL constraints that keep visual and textual representations close to the base model. ALDEN audits only answer accuracy and repetition, never the reasoning text. Point-RFT interleaves coordinate points into reasoning during GRPO (group-relative policy optimization, an RL method that scores each sampled answer against others for the same question). Its reward checks only that the coordinate tags are well-formed, not that they matter. We replace point tags with document region identifiers and replace the format check with quote and necessity checks. A Bounding Box is Worth One Token supplies a compact way to give a language model word-level boxes, so region identifiers cost little context.

Locating Failure in Multi-Page VRDU shapes the input choice. It finds that a hybrid of text, layout, and image (TLV) beats any single representation, and that models fail most when integrating evidence across pages. We use a TLV input and report all results separately for cross-page questions.

Research Program

The input is each fetched page as an image plus recognized words with boxes, grouped into numbered regions (text blocks, table cells, figures). In its reasoning, the model writes steps of the form "region 14 on page 7 says '…'", followed by a claim. For a rollout with answer $a$ on document $D$, citing region set $C$ and quotes $q_1,\dots,q_k$, the reward is

$$r = \mathbb{1}[a \text{ correct}]\Big(1 + \lambda_q \tfrac{1}{k}\textstyle\sum_j m(q_j, C) + \lambda_n \,\mathrm{clip}\big(\Delta(C) - \Delta(R_C)\big)\Big),$$

where $m(q_j, C)\in[0,1]$ is the normalized string overlap between quote $q_j$ and the recognized text of its cited region. $\Delta(S) = \log p(a\mid D) - \log p(a\mid D \text{ with regions } S \text{ masked})$ is the drop in the policy's own answer log-probability when the regions in $S$ are blanked in both image and text. $R_C$ is a random set of regions with the same total area. $\lambda_q$ and $\lambda_n$ are weights.

Gating every term on correctness means the grounding terms can never pay for a wrong answer. Subtracting the random-mask baseline removes the trivial gain from citing huge regions. The necessity term needs one extra teacher-forced forward pass per rollout and no generation. For figure regions, where quote matching does not apply, only necessity is scored. This also gives a direct test of the title-shortcut failure from Plot Twist: if masking a chart's plot area while keeping its title leaves the answer unchanged, the model was reading the title.

The program has two phases. The first is a measurement study. We train an ALDEN-style agent with outcome reward only and log five quantities at each checkpoint: citation necessity, quote accuracy, the hallucination-region rate of Dahiya and Saini (adapted so that "evidence" means the recognized text of the gold pages), legibility on the 1–9 autograder scale of Jose, and the truncation test. If drift does not appear in documents, that negative result is itself worth reporting, and it would redirect the second phase toward the cross-page integration deficit. The second phase trains the grounded reward and its ablations under matched budgets.

Baselines

The naive reference point is the base vision-language model with zero-shot chain-of-thought prompting over the same fetched pages. It shows how faithful an untrained model already is. As anchors, GPT-4o reaches 44.9% F1 on MMLongBench-Doc and human experts 66.0%, per MMLONGBENCH-DOC.

The strongest existing method is ALDEN, retrained by us on the same base model, data (DUDE, MPDocVQA, SlideVQA), number of rollouts, and tuning budget. It is the strongest competitor because it is the only RL document agent in the file and reports a 9.14% average accuracy gain over non-agentic baselines across MMLongBench-Doc, LongDocURL, PaperTab, PaperText, and FetaTab. The file gives no absolute per-benchmark accuracies for ALDEN ("no reported result found on Lacuna" for those), so we compare against our own reproduction. Beating it means accuracy within two points on average together with a clearly higher rate of correct answers whose citations pass both checks. Higher accuracy is not required.

Three further baselines isolate the key ingredient. A Point-RFT-style variant rewards only well-formed region tags. A quote-only variant sets $\lambda_n=0$. A judge-reward variant scores faithfulness with an LLM judge, as Zhao et al. did. A KL-heavy variant raises ALDEN's anchoring strength. All get identical compute, and each sweeps the same number of reward weights.

Why not just…?

Why not just keep the policy close to the base model with a stronger KL penalty? This is the most serious threat, since KL anchoring is how ALDEN stabilizes training. However, KL keeps the output distribution close to the base model, not the evidence. Zhao et al. observed faithfulness drift under standard RL, which normally includes a KL term. Lee et al. found that likelihood-style constraints preserve fluency but still allow meanings to flip. Our KL-heavy baseline settles this. If it matches our method on necessity at equal accuracy, the necessity reward is unnecessary, and we would report that.

Why not just reward faithfulness with an LLM judge? Zhao et al. report that this can collapse training onto shortcuts, and Dahiya and Saini's verifier could not decide 47% of claims. Our judge-reward baseline reproduces this comparison on documents. Our reward uses no judge, so it cannot be fooled by persuasive prose.

Why not just use supervised fine-tuning on human-cited rationales and skip RL? Point-RFT reports 70.88% on ChartQA after format-only supervised tuning, versus 90.04% after RL, along with large out-of-domain gains. Supervised tuning alone also left BakLLaVA's explanations mostly unfaithful. We include a supervised citation model as a baseline. If it matches RL accuracy on long documents, the drift problem becomes moot for this setting.

Why not just attribute after the fact with a separate model? A post-hoc attributor explains the answer but not the computation the policy actually used. Our necessity test measures that computation, and we will report how often post-hoc attributions fail it.

Proposed Experiments

We train on DUDE, MPDocVQA, and SlideVQA, and test on MMLongBench-Doc, LongDocURL, PaperTab, PaperText, and FetaTab, plus DUDE (scored with ANLS, a fuzzy string-match accuracy). We also evaluate robustness under Zhao et al.'s Wrong-Caption and Wrong-Think perturbations: a misleading caption prepended to the input, or an incorrect reasoning prefix injected.

The decisive checks are the following. First, outcome-only RL should show necessity falling across checkpoints while accuracy rises. Second, the quote-only variant should match the full method on quote accuracy but fall well behind it on necessity, which would show decorative citation. Third, the full method should cut the hallucination-region rate relative to ALDEN at near-equal accuracy, with the largest gap on cross-page questions. Fourth, masking cited regions should hurt the full model's answers more than masking random regions in a human-checked sample of 200 items, confirming the reward was not gamed through mask artifacts. Fifth, accuracy drops under Wrong-Caption should shrink, since a model that depends on cited document regions has less reason to trust injected text.