Deployed assistants are almost never asked for a neutral answer. They are asked to write like a clinician, a patent attorney, a financial analyst, or a brand persona, and they usually answer from retrieved documents: retrieval-augmented generation, where passages fetched from a corpus are pasted into the prompt as evidence. We already know two things separately. Asking a model for a particular style can make its answers less truthful even with no documents involved. Separately, irrelevant passages placed among the retrieved documents can pull a model toward wrong answers. This proposal tests a specific way the two problems could combine, which we call register capture. A register is the tone and vocabulary of a text, such as clinical, legal, casual, or archaic. The hypothesis is that a style instruction biases the model toward reading and copying from passages whose register matches the requested output, whether or not those passages contain the answer. Under this hypothesis, a clinically worded but wrong passage would beat a plainly worded correct one whenever the model is told to "answer as a physician."
If register capture is real, it is a failure that current defenses do not target. Hallucination detectors, decoding tricks, and fine-tuning against distractors all assume the output style is fixed and neutral. We propose a controlled benchmark that separates the passage's register from its correctness. We also propose an inference-time test, attention transplant. It copies which documents the model's evidence-reading attention heads look at when the style instruction is absent into the stylized generation, and leaves everything else unchanged. If transplant restores accuracy, the style instruction is causally corrupting evidence selection, not merely the wording. We also commit in advance to a simpler two-stage pipeline as the method to beat.
Core Claim
Our claim is that part of the accuracy loss under stylized generation with retrieved documents happens at evidence selection, not at wording. In mechanistic terms, the loss happens in the small set of attention heads that copy facts from context. This is separate from the model inventing details to sound authoritative, which we treat as a second, independent failure path. The claim is falsifiable in three ways. Register-matched wrong passages may not attract more copying than register-mismatched ones. The model's evidence heads may attend the same way with and without the style instruction. Or transplanting neutral attention may not recover accuracy.
The mechanistic prior work cuts both ways, and each paper plays a different role here. Retrieval Head Mechanistically Explains Long-Context Factuality shows that copying facts from long context runs through fewer than 5% of attention heads. Masking the top 20 of these heads drops needle-in-a-haystack accuracy on Llama-2-7B-80K from about 95% to 63.6%, while masking 20 random heads leaves it at 94.7%. Expert Heads: Robust Evidence Identification for Large Language Models finds 5 to 10 heads that track gold evidence regardless of position, and finds that dispersed attention in them precedes hallucination. These two papers supply our measurement tools and the set of heads we intervene on.
Two further papers motivate the hypothesis. StyliTruth finds style and truth directions entangled in particular heads, but only closed-book, with no documents in the prompt. PrismRAG shows that distractors written in the same style as the gold passage are hard to resist. In PrismRAG, however, the output style never varies and the register of the distractor is never manipulated relative to the requested voice.
The main constraint on our hypothesis is Steering at the Source. It localizes persona to about 3 style heads and reports that steering them leaves knowledge benchmarks (MMLU) and instruction following intact, which suggests style and fact circuits are separate. We take this seriously. That paper tested closed-book multiple-choice questions, where no copying from context is required. Our experiment is the setting that could reconcile or refute its separation claim once context copying is involved.
Research Program
The first component is a factorial benchmark. We start from question sets with known gold passages. These are HotpotQA and MuSiQue in the 2-gold, 8-distractor layout used by the Expert Heads paper, the question-answering portion of RAGTruth (MS MARCO), PubMedQA from RAGBench, and a subset of CRAG. For every question we build a hard distractor in the way PrismRAG does, by mutating the answer-bearing entity in a gold passage. This design lets us trace every answer string back to the passage it was copied from.
We then rewrite passages into registers with an instruction-tuned model while preserving their entities. The rewrites cover clinical, legal/technical, casual, and one literary style (Shakespearean, matching StyliTruth). Each rewrite is checked by two-way natural language inference against the original, and 300 rewrites are checked by hand. The design crosses three factors: the requested output style, the register of the gold passage, and the register of the mutated distractor. The quantity of interest is the register capture rate. It is the increase in the share of answers taken from the mutated distractor when the distractor's register matches the requested style, compared with when it does not, holding everything else fixed.
The second component is mechanistic. For each model we identify retrieval heads with the needle-in-a-haystack retrieval score from Wu et al. and expert heads with the permutation procedure of the Expert Heads paper. We then measure how much of these heads' attention mass falls on the gold passage versus the distractor, with and without the style instruction, token by token through the response. This also lets us test whether style pulls the RAGTruth "tail effect" earlier. The tail effect is the finding that hallucinations cluster toward the end of responses.
The third component is the intervention. At each decoding step we run a second forward pass, the shadow pass. It receives the same retrieved documents and the same generated prefix, but a style-neutral instruction. For each head $h$ in the identified set $R$, we replace the stylized pass's attention over document positions $D$ with the shadow pass's distribution. We rescale it so that the total attention on documents stays unchanged:
$$ \tilde a_h(j) = a^{\text{neu}}_h(j)\,\frac{\sum_{k\in D} a^{\text{sty}}_h(k)}{\sum_{k\in D} a^{\text{neu}}_h(k)}, \quad j \in D, $$
Here $a^{\text{sty}}_h(j)$ and $a^{\text{neu}}_h(j)$ are head $h$'s attention weights on document token $j$ in the stylized and neutral passes. Attention on non-document tokens is left as it is. The rescaling keeps "how much to consult the documents" under the style prompt's control and hands only "which document" to the neutral pass. Transplant costs one extra forward pass per token, the same overhead as the contrastive decoding in DeCoRe, and it requires no training and no contrast datasets.
We run Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct (where the style heads were located), and Qwen3-8B with thinking mode on and off. The Qwen3-8B condition tests whether register capture compounds the "rationalization trap" reported in Thinking in a Crowd. In that trap, two misleading snippets drop accuracy from 96.22% to 77.03% with thinking on, worse than the 74.87% with thinking off.
Baselines
The naive reference point is stylized generation with retrieval, where the style instruction and all retrieved passages simply go in one prompt. We run it next to the same prompt without a style instruction, which gives the neutral ceiling. The gap between the two is the quantity every method tries to close. For scale, CRAG reports about 44% accuracy for straightforward retrieval-augmented generation (CRAG). No reported result was found on Lacuna for any retrieval benchmark under a requested output style, so the stylized numbers here would be new measurements.
The strongest competitor is filter-then-style. A style-neutral pass first selects evidence, and the stylized generator then sees only the selected passages. We run two versions. The first uses Expert Head Voting, which reports 90.72% Precision@2 on HotpotQA against 75.23% for a dense BGE retriever (Expert Heads). The second uses neutral-instruction sentence extraction modeled on Ext2Gen. Ext2Gen reports an average QA score of 0.557 against 0.473 for the default Llama3.1-8B model. We use its extract-then-generate structure without its preference training, so that compute stays matched. Filter-then-style is the strongest competitor because it removes the style instruction from selection entirely.
A second competitor is neutral-then-restyle: answer neutrally, then ask the model to rewrite the answer in the target style.
We also compare against DeCoRe, the strongest training-free decoding method in the file. It contrasts the model against a copy with its retrieval heads masked and reports a 5.5% exact-match gain on NQ-Swap and 18.6% on XSum factual consistency.
For style control we compare prompting against steering at the style heads from Steering at the Source. This checks whether register capture is specific to prompting or also appears under steering.
All methods use the same retrieved sets, the same models, and greedy decoding. Every tunable setting, including DeCoRe's entropy scaling, the filter's top-k, and the choice of transplant head set, gets the same budget of 20 configurations on a held-out development split.
Two ablations isolate the key ingredient. The first transplants attention in an equal number of random heads, which tests whether the specific heads matter. The second transplants in the identified heads but over non-document positions only, which tests whether it is document selection that matters.
To count as beating the baselines, transplant must close a significantly larger fraction of the stylized-to-neutral accuracy gap than filter-then-style (paired bootstrap, p<0.05, on at least two of three models). It must also keep style-classifier strength within 5% of the naive stylized run.
Why not just…?
Why not just tell the model to stick to the evidence and ignore distractors? Prompting mostly changes tone, not grounding. GPT-4V Cannot Generate Radiology Reports Yet found that advanced prompting improved report formatting while leaving predictions statistically independent of the true findings. Self-Knowledge Guided Retrieval Augmentation found direct prompting to be the weakest way to get models to recognize what they do not know. We still include a strict-grounding instruction in every condition, so any register capture we measure is what remains after that instruction.
Why not just answer neutrally and restyle afterward? The rewrite step is the kind of stylization that the fine-tuning literature finds prone to invention. Ghosh et al. and Knappich et al. show models adding details to satisfy a format. The rewriter also no longer sees the evidence. We expect restyling to fix evidence selection but to add baseless content, which we measure with a RAGTruth-style span detector. That detector type reaches 78.7% response-level F1, against 63.4% for prompted GPT-4 (RAGTruth). If restyling adds no more baseless spans than transplant, it wins on simplicity and we will say so.
Why not just filter evidence neutrally, then generate in style? This is the real threat. If register capture lives entirely in passage-level selection, filtering captures nearly all of the benefit, and at lower cost. Transplant can only win where the choice among evidence is finer-grained or shifts over the course of generation. Examples are several gold passages that each contribute part of a multi-hop answer, a distractor that shares sentences with gold, or drift late in long responses. Our factorial design includes these cases. If filter-then-style matches transplant, the mechanistic finding still stands, but we will recommend filter-then-style as the practical fix and present transplant as the diagnostic that showed why it works.
Why not just use DeCoRe, or fine-tune against distractors as PrismRAG does? DeCoRe's weak model has retrieval heads masked but keeps the style instruction. When the two are contrasted, stylistic tokens that appear nowhere in the documents may be penalized as ungrounded, and a register-matched distractor is itself context that the contrast would reward. We test both effects directly. PrismRAG reports a 5.4% average factuality gain over Llama-3.1-70B-Instruct, but it trains on distractors in the same register as gold. A model fine-tuned that way under our budget is a fair comparison on the mismatched-register cells. If it also resists register-matched distractors, fine-tuning is an adequate fix for deployments that can afford to train.
Decisive Checks
The first check is existence. We ask whether the register capture rate exceeds zero, with a 95% bootstrap interval, on at least two datasets and two models. A null result here is informative: it would extend the Steering at the Source finding, that style is separate from fact circuits, to the setting where facts must be copied from context. We would publish that as the finding.
The second check is mechanism. We ask whether evidence-head attention on the style-matched distractor rises under the style instruction, and whether that rise predicts which answers are copied from the distractor, question by question.
The third check is causality. Transplant in the identified heads should recover accuracy, while transplant in random heads, or over non-document positions, should not.
The fourth check is the practical comparison with filter-then-style, which we have committed to report either way.
Secondary analyses ask whether style shifts the RAGTruth tail effect earlier in responses and whether thinking mode amplifies register capture. Both cost little on top of the main runs, because the same outputs support them.
References
- Retrieval Head Mechanistically Explains Long-Context Factuality
- Expert Heads: Robust Evidence Identification for Large Language Models
- StyliTruth : Unlocking Stylized yet Truthful LLM Generation via Disentangled Steering
- PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoning
- Steering at the Source: Style Modulation Heads for Robust Persona Control
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- CRAG - Comprehensive RAG Benchmark
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations
- GPT-4V Cannot Generate Radiology Reports Yet
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models
- A Closer Look at the Limitations of Instruction Tuning
- PAP2PAT: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs
- RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models