Masking the Unsupported: Token-Level Loss Masking for Fine-Tuning on Fragmented Context

When a language model is fine-tuned to answer questions from retrieved passages, the passage it sees is usually a fragment of a larger document. Chunking has cut away headers, earlier definitions, and the sentence where an entity was introduced. The gold answer was written by someone who had the whole document. As a result, some answer tokens are supported by the visible fragment, some the model already knows from pretraining, and some are justified only by text the model never saw. Standard fine-tuning puts equal gradient on all three kinds. For the third kind, that means rewarding the model for producing content its input does not justify, which is close to the definition of hallucination.

We propose to find these unsupported tokens automatically and remove them from the loss. Each answer token is scored by comparing a frozen reference model's prediction in three conditions: with no context, with the fragment, and with the full document. A token that becomes predictable only once the severed text is restored gets zero loss. It still appears as input to later positions, but the model is never pushed to generate it. The idea is simple: fine-tuning should only reward outputs that are justified by the input actually shown, or by what the model already knows. We test it on dynamically generated multi-hop documents, where we control exactly which premise is cut and where memorized benchmark answers cannot confound the result.

Core Claim

Our claim is that part of the hallucination produced by retrieval-grounded fine-tuning comes from the training signal, not from the model: gradients placed on answer tokens whose support was severed by chunking. Masking exactly those tokens should lower hallucination under fragmented context while keeping two things intact. The first is closed-book knowledge. The second is the ability to answer when the fragment is sufficient. The masking has to be token-level and counterfactual, so it cannot be replaced by dropping whole examples or by masking the context.

The closest existing uses of loss masking each mask something different. Context-Informed Grounding Supervision (CINGS) masks the prepended context and supervises every response token. That shifts over 60% of attention onto the context, but it assumes the context is complete, and the authors report that models trained this way lose parametric knowledge on closed-book tasks. Do I Really Know? (VeriFY) masks whole stages of self-verification traces that contain hallucinated intermediate answers, on short-form QA with no retrieved context. Empirical Evaluation of Loss Masking to Selectively Prevent Memorization masks predefined sensitive substrings.

Our masked set is instead computed per token, from whether the visible input supports that token. That is the specific failure chunking creates. The support signal borrows the verification idea of EntropyLong, which keeps a retrieved chunk only if prepending it measurably lowers a target token's entropy. EntropyLong uses this signal to select data and then trains with uniform loss. We turn the same kind of signal into the loss mask.

Research Program

Support score. For a training example with fragment $c_f$, full source document $c_d$, and gold answer tokens $y_1,\dots,y_T$, let $\ell_t(c) = \log p_0(y_t \mid c, y_{<t})$ be the frozen reference model's log-probability of token $t$ given context $c$. The empty context is written $\varnothing$. We compute two quantities:

$$g^{\text{frag}}_t = \ell_t(c_f) - \ell_t(\varnothing), \qquad g^{\text{sev}}_t = \ell_t(c_d) - \ell_t(c_f).$$

Here $g^{\text{frag}}_t$ measures how much the visible fragment helps predict token $t$. $g^{\text{sev}}_t$ measures how much the missing text would additionally help. A token is masked ($m_t = 0$) when $g^{\text{sev}}_t > \tau$ and $\ell_t(c_f)$ stays below a confidence floor. In words, the evidence for it lives in the cut-away text and neither the fragment nor prior knowledge makes it predictable. Every other token keeps $m_t = 1$, and the fine-tuned model $p_\theta$ is trained on

$$\mathcal{L} = -\frac{1}{\sum_t m_t}\sum_t m_t \log p_\theta(y_t \mid c_f, y_{<t}).$$

Tokens that are predictable without any context are deliberately left supervised. This is our bet on the CINGS trade-off: the model should not be discouraged from using what it already knows, only from producing what nothing supports.

A controlled fragmentation testbed. Estimating support is only trustworthy if we can check it against ground truth. Following DyVal, which generates reasoning problems on the fly from directed acyclic graphs, we generate multi-hop fact graphs. Following INVITE, the facts are perturbed DBpedia triples, so they cannot be recalled from pretraining. Each graph is rendered as a document with section structure, coreference, and scoped conditions, then chunked so that chosen premise nodes fall outside the fragment.

Because we generate the graph, we know which answer tokens depend on which severed premise. That lets us measure the estimator's precision and recall on unsupported tokens directly, and choose $\tau$ on held-out graphs rather than by guesswork. EntropyLong reports exactly this threshold choice as its main unresolved difficulty. Graph depth and the order of premise presentation are varied, since DyVal finds depth and reversed order the most damaging factors: GPT-4 drops 13.6% on reachability under reversal.

Benchmark contamination as a measurement constraint. The no-context term $\ell_t(\varnothing)$ also flags memorized answers. On static sets such as HotpotQA and NQ-Open, we report the fraction of gold answer tokens the base model predicts closed-book. We report hallucination separately on high- and low-memorization strata. This keeps a masking gain from being really a gain on leaked items, a concern the contamination literature documents extensively: models strong on static benchmarks score near zero on DyVal. Our main claims rest on the dynamic testbed. Static sets serve as transfer checks.

Natural data. We then apply the method to realistic fragmentation: multi-hop QA whose source documents are chunked by a standard fixed-window retriever. For this setting, the full document $c_d$ is simply the unchunked source. Models come from the Llama, Qwen, and Gemma families at roughly the 7–9B scale, matching VeriFY. All methods use identical data, steps, and learning-rate sweeps.

Baselines

The naive reference point is uniform-loss fine-tuning: the same fragment–answer pairs, trained with standard cross-entropy over the whole sequence, context included. This is how DyVal's fine-tuning was done and how EntropyLong trains. It measures how much hallucination the unfiltered training signal causes.

The strongest existing competitor is CINGS (Context-Informed Grounding Supervision): response-only loss with the context masked, trained on exactly our fragments. It is the strongest because it already targets grounding through loss masking and reports reduced hallucination on 11 information-seeking benchmarks. Our method differs from it only in which response tokens receive gradient. CINGS reports an 18.1% improvement on LLaVA-Wild under strict evaluation, but no reported result was found on Lacuna for CINGS on fragmented-context or dynamically generated multi-hop QA. We will run it ourselves under the same tuning budget. Beating it means lower hallucination on the fragmented testbed at equal or better accuracy on sufficient-fragment items, plus smaller closed-book loss than CINGS.

The second strong competitor is VeriFY (Do I Really Know?), retrained on our data with its stage-level masking. It reports hallucination reductions of 9.7 to 53.3 percent at a recall cost of 0.4 to 5.7 percent on TriviaQA-trained models evaluated on HotpotQA and NQ-Open. The fair comparison is on those same out-of-domain sets and on our testbed. Matching VeriFY's recall cost while reducing hallucination more under fragmentation would count as beating it. Combining the two is also a natural arm.

Three ablations isolate the key ingredient. Random masking masks the same fraction of tokens chosen at random. Oracle masking uses the testbed's ground-truth dependencies and gives an upper bound on what a better estimator could buy. Fragment-only scoring drops the no-context term and masks every token the fragment does not support, including memorized ones. That tests whether keeping parametric tokens supervised is what protects closed-book performance.

Why not just…?

Why not just drop training examples whose fragment does not contain the answer? This is the most serious threat, and it may work nearly as well. Example-level filtering discards every token of a partially supported answer, including the supported reasoning and phrasing. On multi-hop data, where almost every fragment is missing something, it may discard most of the data. We run it at matched compute. If it matches our hallucination and accuracy numbers, the token-level machinery is unnecessary, and we will say so. The dose-response experiment on graph depth below is designed to show where the two diverge: as depth grows, the share of partially supported examples rises.

Why not just relabel severed-context examples with an "insufficient information" target? Relabeling teaches abstention, but it throws away the supported part of the answer. It also risks the over-refusal that VeriFY reports for knowledge-probing baselines, which became conservative and lost many correct answers. Masking is a different intervention: it removes a harmful gradient without adding a new behavior. The two can be combined, and we include relabeling both alone and combined with masking. We expect masking alone not to produce explicit abstention, so AbstentionBench is where relabeling should win and where the combination has to justify itself.

Why not just give the model the full document, via long-context windows or late chunking? Late Chunking and long-context training reduce fragmentation at retrieval time. At deployment, however, retrieval still returns partial evidence, and EntropyLong itself notes lost-in-the-middle effects without special training. Our method needs the full document only at training time, to score tokens. We add a full-document fine-tuning arm, evaluated under fragmented inference, to test whether the mismatch between training and inference context matters.

Why not just prompt the model to verify its conditions at inference? The evidence argues against expecting this to be enough. AbstentionBench finds reasoning models 24% worse at abstaining, with more test-time compute making it worse. Can Tool-augmented Large Language Models be Aware of Incomplete Conditions? shows a Structured Verification checklist lifting Gemma-3-4B from 20.90% to 68.57% F1 on ToolBench API Replacement, still below the roughly 81% human baseline. We apply Structured Verification prompting on top of every trained model, so any gain from masking is measured beyond what prompting provides.

Proposed Experiments and Decisive Checks

The first check validates the estimator. On the testbed, its precision and recall against ground-truth unsupported tokens must clearly exceed chance, and they must hold as depth and presentation order vary. If the estimator fails here, the method is not validated.

The main experiment measures three things for each method on held-out dynamic graphs. The first is the hallucination rate on items whose fragment is insufficient, judged by exact graph-derived answers so that the BLEU/ROUGE blindness INVITE documents does not apply. The second is accuracy on items whose fragment is sufficient. The third is closed-book accuracy on TriviaQA and MMLU, which tracks the loss of parametric knowledge.

The decisive pattern is specific. Our method should beat CINGS and uniform loss on hallucination and beat CINGS on closed-book retention. Random masking should not reproduce the gain. Oracle masking should bound it from above. The gap over example-level filtering should widen with graph depth.

Transfer runs on HotpotQA and NQ-Open are split by memorization stratum. They also cover the Underspecified Context split of AbstentionBench, where masking alone versus masking plus relabeling shows whether removing gradient is enough without teaching abstention. Finally, the Utterance Removal split of ICT tests transfer to missing user input.

One constraint deserves a direct test. The memorization study found that masked tokens, although never emitted, still taught the downstream mapping: backdoor functionality rose to 89% from 75% unmasked. The same leakage could happen here. Teacher forcing feeds unsupported gold tokens as input to later positions, so the model may learn to condition on facts it was never shown. We therefore measure whether tokens after a masked span still reproduce the severed fact when the model generates freely. If they do, we will extend the mask to cover tokens that depend on the masked span. Whether that extension is needed is one of the results.