A retrieval-augmented language model is a model that answers questions using documents a search system fetches for it. It often needs to say "I don't know" when those documents do not actually contain the answer. The standard way to teach this, called abstention training, pairs questions with obviously unrelated documents and trains the model to refuse. In technical domains that lesson is too easy. When a financial analyst asks for 2021 operating margin, the dangerous document is not a cooking recipe. It is the same company's report with a table covering 2022 and 2023. When an engineer asks for a library function, the dangerous context is that library's documentation, which lists fifty similar functions but not the requested one. These near-miss contexts share vocabulary, layout and numbers with the true evidence. Our claim is that models trained on random or merely retrieved negatives learn to spot a topic mismatch, not to check whether the evidence is sufficient. We also expect small models specialized on a domain to fail worst on near-misses, because their dense domain memory makes it easy to fill in the one missing fact.
We propose building abstention training data from minimal perturbations. We take an answerable question and its real context and change the single attribute that ties them together, such as the year, the entity, the unit, the table row or the function name. The context is then almost right but no longer supports an answer. Each negative comes paired with its answerable twin, which differs by one token-level edit. The model is therefore trained to refuse and to not over-refuse on the same inputs. The location of the edit also tells us, without human annotation, which span of the context the decision hinges on. That removes the sentence-level labeling bottleneck that limits current reinforcement-learning approaches to grounded abstention.
Core Claim
The claim is that the type of negative context matters more than the training algorithm for abstention in specialized domains. We test it by holding the trainer fixed and varying only where the negatives come from. There are three conditions: random unrelated documents, in-domain retrieved documents with the gold passage removed, and minimal-edit near-misses. We predict three results. Minimal-edit negatives will raise abstention recall on natural, human-made unanswerable questions in finance, science and API domains. They will do so without lowering accuracy on answerable ones. And the gain will be largest for domain-specialized backbones.
Relation to Prior Work
Three existing papers motivate the gap by showing that current specialized models do not abstain. RAFT fine-tunes models on a gold document mixed with distractor documents, sometimes dropping the gold document, and it adapts well to PubMed and API documentation. Divide-Then-Align shows that RAFT-style models reach 0.0% abstain recall because they are trained to always answer. EC-RAFT carries the RAFT recipe into clinical-trial design with no refusal path at all. We start from the RAFT regime of distractor-aware domain specialization and add the missing abstention behavior.
Two papers supply the training method we reuse. Divide-Then-Align sorts training examples into four groups according to whether the model's own memory knows the answer and whether the retrieved documents contain it. It then applies direct preference optimization (DPO), which trains the model to prefer a chosen response over a rejected one, with "I don't know" as the chosen response when neither source has the answer. It reports 64.7% abstain F1 for LLaMA-3-8B on WebQuestions. GRACE uses reinforcement learning with a gated reward that checks format, then the decision to answer or abstain, then content accuracy. On QASPER it lifts the unanswerable score from 42.23% to 74.90%, but it needs gold evidence sentences for the content reward. Our proposal keeps both training algorithms fixed and changes their inputs. Both papers draw negatives from general-domain retrieval, where unanswerable contexts are usually easy to separate. Neither tests domain-adapted backbones.
Several older lines build negatives in ways we deliberately depart from, and together they constrain our design. On Controlling Fallback Responses for Grounded Dialogue Generation and Zero-Shot Dialogue State Tracking via Cross-Task Transfer pair queries with unrelated documents, which is the "random negatives" condition we expect to underperform. Robust Function-Calling via Function Masking removes the relevant tool from the schema. That is already a near-miss in the API setting, and we generalize the idea to text and tables. Recall with Reasoning cuts answer spans out of documents. We avoid heavy context editing because it can leave traces a model learns to detect: a missing sentence, a broken table. Our main perturbation therefore edits the question and leaves the real document untouched. RbFT is a negative result that constrains us. It trains models to classify retrieved documents as noisy, irrelevant or counterfactual, and training on defect detection alone hurts performance on clean retrieval. That is why every near-miss negative in our data is paired with its answerable twin.
Two papers show why these domains are hard. UDA shows that numbers buried inside financial reports defeat long-context models. ESG-Bench reports up to 99.5% accuracy on its "Without Answer" cases after fine-tuning. We read that near-ceiling figure as a sign that its unanswerable cases are too easy, not as evidence that the problem is solved. CMExam reports that specialized medical models copy expert formats but hallucinate factual details. That observation is the basis for our hypothesis that specialization makes near-miss failures worse.
Research Program
We build near-miss generators for three domains, chosen because each has an obvious attribute that ties the question to the evidence. In finance we use UDA's FinHybrid and TatHybrid, where the tying attributes are the fiscal period, the line item, the segment and the unit. In scientific papers we use QASPER and UDA's PaperTab and PaperText, where they are the dataset name, the metric and the model variant. In API documentation we use Gorilla's HuggingFace, Torch Hub and TensorFlow Hub splits from RAFT, where they are the function or model name and the task.
For each answerable example, a rule-based perturber proposes substitutes that stay in the domain's vocabulary. Examples are a year adjacent to the covered range, a sibling line item, or a sibling function drawn from the same library. A checker, made of string and table matching plus an agreement vote from a strong model acting as judge, confirms that the perturbed question is unanswerable from the context. A smaller share of negatives edit the context instead, for example by deleting one table row. We keep this share small and measure separately whether it introduces detectable artifacts.
Each edit comes with a localization for free. The span that answered the original question is the span that nearly matches the perturbed one. This gives GRACE-style reinforcement learning a weak evidence label. The reward for abstaining can require the model to cite that near-miss span as the reason it refused, and no human span annotation is needed.
We train two families of backbones, general instruction models (Llama-3.1-8B-Instruct and Qwen3-4B) and domain-specialized versions of the same models. We create the specialized versions ourselves by RAFT-style fine-tuning on each domain's training split, so the only difference between a general model and its specialized counterpart is the specialization step. Onto each backbone we apply abstention training with Divide-Then-Align's DPO recipe, which is the primary setting, and with GRACE's gated reinforcement learning, which is secondary. In each case we vary only the source of the negatives.
The analysis has two parts. First, we test whether specialization shifts examples from the "retrieved documents lack the answer, model does not know it" group into confident hallucination. We do this by measuring how often the model answers near-miss questions from memory, before and after specialization. Second, we test whether the gains transfer to unanswerable questions that no perturber of ours produced.
Baselines
The naive reference point is each backbone prompted with an explicit abstention instruction, both before and after RAFT specialization. It shows what instruction-following alone buys. The RAFT-specialized model without abstention training is the regime whose failure motivates this work, and Divide-Then-Align reports 0.0% abstain recall for RAFT-style models.
The strongest existing method on this question is Divide-Then-Align run with its own negative construction, meaning general retrieval where the gold passage is withheld. It reports 64.7% abstain F1 for LLaMA-3-8B on WebQuestions. GRACE is the strongest reinforcement-learning competitor. The file reports 74.36% overall accuracy and 74.90% on unanswerable questions for GRACE-Qwen3-4B on QASPER, and 81.20% overall accuracy on HotpotQA. We found no reported abstention result on Lacuna for UDA, for the Gorilla API splits, or for any domain-specialized backbone. We therefore rerun both methods ourselves on our domains. For every method we use the same backbones, the same number of training pairs (around 10,000, following Divide-Then-Align), the same number of optimizer steps and the same hyperparameter-search budget. We also sanity-check that our reimplementation roughly reproduces GRACE's QASPER unanswerable figure. We count as beating these methods a gain in abstain F1 on the natural unanswerable test sets that holds across three training seeds, together with answerable accuracy no more than one point below the best baseline.
The ablations that isolate the key ingredient keep the trainer and pair count fixed and swap the source of the negatives. The first swap uses random unrelated documents. The second uses in-domain retrieved hard negatives, meaning the top BM25 or dense-retrieval results with the gold passage removed. This is the most important ablation. A further ablation drops the paired answerable twin, which tests whether pairing is what prevents over-refusal. A last one compares question-side edits against context-side edits, to check for artifacts introduced by editing documents.
Why not just…?
Why not just prompt the model to say "I don't know" when unsure? ESG-Bench reports that standard prompting reaches 76% on LLaMA-3.2, against 96% for its fine-tuned structured reasoning. Divide-Then-Align shows that fine-tuned models ignore abstention instructions entirely. We keep prompting as a baseline anyway. It is cheap, and on near-misses it is not obvious that trained methods beat it.
Why not just use in-domain retrieved documents as the hard negatives? This is the real threat. Retrieved passages without the gold document are already in-domain and look plausible, and they cost nothing to build. Our reason for expecting them to fall short is that they usually differ from the gold passage in many attributes at once, so a model can reject them through topical drift without checking sufficiency. We do not argue this point; the retrieved-negative ablation, run at equal pair count, settles it. If retrieved negatives match minimal edits on the natural unanswerable test sets, the core claim fails. The contribution then reduces to the diagnostic result about domain specialization, and we would report it that way.
Why not just put a separate answerability classifier in front of the generator? A classifier that checks whether the passage entails an answer to the question can gate the model without retraining it. Ext2Gen finds, however, that separate pre-generation filters such as Recomp, CompAct and EXIT lose to extraction inside generation, and they add latency. We will still train such a classifier on exactly our near-miss data and use it as a gate. If it matches the end-to-end model, that would suggest the value lies in the data rather than in training the generator. We would consider that an acceptable and useful outcome.
Why not just let the specialized model read the whole document? UDA reports that on FinHybrid, long-context GPT-4 without retrieval scores 0.4, against 45.9 with retrieval. Longer context adds near-miss content rather than removing it.
Proposed Experiments
The primary evaluation uses unanswerable questions that no generator of ours produced. These are QASPER's native unanswerable questions, ESG-Bench's "Without Answer" cases, IncompleteToolBench for missing tools and incomplete user requests, and a set of about 300 near-miss questions per domain written by humans over UDA financial reports and Gorilla documentation. Annotators write these without seeing our perturbation rules. We report answerable accuracy using each benchmark's native metric, plus abstain precision, recall and F1, and an over-refusal rate on answerable twins. Because our training perturbations and these tests come from different sources, a model cannot win by learning our edit patterns.
The first decisive check is the ablation over negative sources on general and specialized backbones. The claim holds only if minimal-edit negatives beat retrieved hard negatives on the human-made near-miss set without increasing over-refusal. The second check tests the specialization hypothesis. Before abstention training, RAFT-specialized backbones should hallucinate on near-miss questions more often than their general counterparts. After training with minimal-edit negatives, that gap should narrow more than it does under the other negative sources. If specialization does not increase near-miss hallucination, we drop that part of the claim.
The third check is the weak-span reward. It tests whether GRACE trained with edit-derived localization gets within a few points of GRACE trained with gold QASPER evidence sentences. That comparison is only possible on QASPER, where gold evidence exists, and it determines whether the method extends to domains without span annotations, such as UDA and clinical trials. A final artifact check trains a probe to distinguish context-edited documents from real ones. If the probe succeeds well above chance, we limit training to question-side edits.
References
- RAFT: Adapting Language Model to Domain Specific RAG
- Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG
- EC-RAFT: Automated Generation of Clinical Trial Eligibility Criteria through Retrieval-Augmented Fine-Tuning
- GRACE: Reinforcement Learning for Grounded Response and Abstention under Contextual Evidence
- On Controlling Fallback Responses for Grounded Dialogue Generation
- Zero-Shot Dialogue State Tracking via Cross-Task Transfer
- Robust Function-Calling for On-Device Language Model via Function Masking
- Recall with Reasoning: Chain-of-Thought Distillation for Mamba’s Long-Context Memory and Extrapolation
- Robust Fine-tuning for Retrieval Augmented Generation against Retrieval Defects
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- ESG-Bench: Benchmarking Long-Context ESG Reports for Hallucination Mitigation
- Benchmarking Large Language Models on CMExam - A comprehensive Chinese Medical Exam Dataset
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?