A common way to make a large language model classify better without training it is to show it a few labeled examples in the prompt, each with a short explanation of why the label is right. This is in-context learning: the model infers the task from examples in its input and never updates its weights. Often a model writes those explanations itself. But models invent facts, and a self-written explanation can contain a confident, fabricated claim that happens to support the correct label. Nobody has measured what such a poisoned explanation does to the classifier that learns from it. We propose to measure it, to trace it inside the model, and to test a cheap fix: explanations built only from verbatim quotes of the input. A quote can be checked against the source by exact string matching, so fabrication becomes detectable at no cost.
The question matters because the evidence on in-context explanations conflicts. One empirical study found that prepending generated explanations to zero-shot prompts lowers large-model accuracy by 7% to 15% on e-SNLI, a benchmark where a model decides whether one sentence follows from, contradicts, or is neutral to another (Can LLM-Generated Textual Explanations Enhance Model Classification Performance?). AnnoLLM reports the opposite in the few-shot setting: self-generated rationales raised GPT-3.5 accuracy on a query-keyword relevance task (QK) from 67.80% to 75.60% (AnnoLLM). Our hypothesis is that part of this disagreement comes from how much ungrounded content the rationales carry, and that this can be measured directly.
Core Claim
We claim two things about frozen models doing few-shot classification with rationale-bearing examples (demonstrations). First, when a rationale contains a fabricated factual claim, accuracy drops more than it would with no rationale at all, even though the label is correct. Second, the damage is visible in the model's internal "task vector." Prior work showed that in-context learning compresses the demonstrations into one hidden-state vector at an intermediate layer, and that patching this vector into a query-only forward pass recovers 80-90% of normal in-context accuracy (In-Context Learning Creates Task Vectors). We predict that rationales anchored in quotes produce cleaner, more transferable task vectors than free-text rationales of equal correctness.
What is new is the controlled study of hallucination inside demonstration rationales, combined with a mechanistic readout. AnnoLLM and PINTO treat prompted rationales as trustworthy scaffolding. PINTO trains a small model to rely on rationales produced by a 20B prompted model, but it never checks those rationales for false premises. On the hallucination side, the strongest grounding results come from training. RLSeek uses reinforcement learning with a penalty for unfaithful quotations, and Copy-Paste uses preference fine-tuning to make models copy from context. We ask whether the quoting constraint works purely through demonstrations, with no weight updates, and why.
Research Program
The first piece is a factorial set of demonstrations built for each benchmark. Each demonstration keeps its correct label and varies along two axes. The first axis is the rationale's form: none, free-text, or quote-anchored. A quote-anchored rationale is a sequence of verbatim spans from the input joined by short fixed connectives such as "therefore" or "this contradicts." The second axis is the rationale's factual status: faithful, or containing exactly one fabricated claim that still supports the label.
The fabricated variant takes a different form for each rationale type. For free-text rationales it is an invented background fact. For quote-anchored rationales it is a fake quote that does not occur in the input. This construction makes the cheap fix testable: an exact string match should catch every fake quote. Faithful free-text rationales come in two kinds, model-generated in the AnnoLLM manner and human-written where gold explanations exist, as in e-SNLI.
The second piece is the mechanistic readout. We use open models whose internals we can read, Qwen2.5-7B and Llama-3.1-8B. At the query, the model outputs only a single-token label, which keeps us within the single-token regime where task vectors are well established. For each condition we extract the task vector at the best intermediate layer, which we locate the same way the task-vector paper does. We then measure three things: how much of that condition's accuracy the patched vector recovers, how far the vectors from corrupted and faithful demonstrations sit from each other, and whether patching a faithful-condition vector into a corrupted prompt restores accuracy. The task-vector paper showed that a patched vector overrides conflicting prompt text. If that holds here, the patch would show the damage passes through this vector rather than through direct attention to the false tokens.
The third piece applies the method to hallucination detection, framed as classification on RAGTruth (RAGTruth). Here the input is a source document plus a generated response, and the label says whether the response contains unsupported content. In-context demonstrations carry quote-anchored rationales that cite the source, which gives an in-context counterpart to RLSeek's training-time quotation penalty.
We expect a boundary condition on HealthFC, a claim-verification set where explanations helped zero-shot models because they supplied outside medical knowledge. Anchoring only to the input should hurt there, because it removes exactly that knowledge. Our test of this is to anchor to retrieved evidence instead.
Baselines
The naive reference is plain few-shot prompting with labels only and the same demonstrations, along with zero-shot prompting. Every condition uses identical demonstration inputs, identical label sets, and the same eight random demonstration orderings. Order matters because in-context accuracy is known to be sensitive to phrasing and ordering (Zero- and Few-Shot NLP with Pretrained Language Models). We report the mean and the spread across orderings.
The strongest in-context competitor on the core question is AnnoLLM's explain-then-annotate method. It reports 75.60% on QK against 67.80% for plain few-shot GPT-3.5, and we will rerun it on our models with the same demonstrations. Beating it means that quote-anchored demonstrations match or exceed its mean accuracy and show a smaller spread across orderings.
For the application, the strongest prompted reference is GPT-4 on RAGTruth, which reaches 63.4% response-level F1 and 28.3% span-level F1. Trained detectors set the ceiling we do not expect to reach with a frozen 8B model: fine-tuned Llama-2-13B reaches 78.7% response-level F1 (RAGTruth), and RL4HS-7B reaches 55.9 span-F1 (Learning to Reason for Hallucination Span Detection). We count success as exceeding prompted GPT-4's response-level F1 with a frozen 8B model. Closing part of the gap to the trained detectors would be a bonus, not the test.
On e-SNLI we found no reported absolute few-shot LLM result on Lacuna; the cited study reports only the 7-15% zero-shot drop. We will therefore establish all e-SNLI numbers ourselves under one protocol.
The ablation that isolates our key ingredient is the factorial itself. Comparing quote-anchored rationales against gold human free-text rationales holds correctness fixed and changes only form. Comparing faithful against fabricated variants holds form fixed and changes only correctness.
Why not just…?
Why not just drop rationales and use label-only demonstrations? Rationales do sometimes hurt, and the empirical study's zero-shot drop is a real warning. AnnoLLM's few-shot gain suggests the effect depends on the regime, however, and label-only prompting is one arm of our design. If label-only matches quote-anchored demonstrations on every benchmark, we will report that rationales add nothing in-context for these models. That would be a useful negative result.
Why not just filter generated rationales with a hallucination detector? Detectors are imperfect. GPT-4 reaches only 28.3% span-level F1 on RAGTruth, so a filter would let many fabricated claims through and add a second model call. Anchoring turns verification into exact string matching. We will still run a filtered AnnoLLM arm, using RAGTruth-style judging by the same frozen model. If filtering matches anchoring, the contribution narrows to cost and simplicity, and we will say so.
Why not just use correct rationales, such as human gold ones? This is the most serious threat, because quoting might matter only as a proxy for being correct. The gold free-text against quote-anchored comparison settles it directly. If the two are equal in both accuracy and task-vector quality, then the form of the rationale adds nothing, and the paper's claim reduces to "rationale correctness matters, and quotes are a cheap way to certify it."
Why not just ask the model to self-correct its rationales? Intrinsic self-correction has collapsed elsewhere: on QAMPARI, correctness recall fell from 18.6 to 4.8 (Ask, Assess, and Refine). We include one self-refine arm for completeness but expect it to underperform.
Why not just train a probe on hidden states instead of studying task vectors? FACTUM's probes detect citation hallucination from internal signals, with a peak AUC of 0.737 (FACTUM). They diagnose output tokens, though, and do not explain how demonstrations shape the classifier. We use task vectors as an explanatory instrument, not as a competing detector.
Proposed Experiments
The benchmarks are e-SNLI, where the large-model utility gap was found; HealthFC as the knowledge-dependent boundary case; QK and BoolQ to compare with AnnoLLM; the FaithEval counterfactual subset to test robustness to conflicting context, where GPT-4o scores 47.5% (Copy-Paste); and RAGTruth at the response level.
We measure accuracy or macro F1, the spread across demonstration orderings, the fraction of accuracy a patched task vector recovers, and the share of fabricated quotes caught by string match.
Three checks decide the outcome. First, if fabricated free-text rationales do not lower accuracy relative to faithful ones by more than the ordering spread, the poisoning hypothesis fails. Second, if quote-anchored rationales do not beat gold free-text rationales, anchoring has no effect beyond correctness. Third, if patching a faithful-condition task vector into a corrupted prompt does not restore at least half of the lost accuracy, the damage does not pass mainly through the task vector, and the mechanistic account is wrong for this setting.
References
- Can LLM-Generated Textual Explanations Enhance Model Classification Performance? An Empirical Study
- AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
- In-Context Learning Creates Task Vectors
- PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales
- RLSeek: Evidence-Grounded Reasoning for RAG Hallucination Detection
- Copy-Paste to Mitigate Large Language Model Hallucinations
- RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
- Zero- and Few-Shot NLP with Pretrained Language Models
- Learning to Reason for Hallucination Span Detection
- Ask, Assess, and Refine: Rectifying Factual Consistency and Hallucination in LLMs with Metric-Guided Feedback Learning
- FACTUM: Mechanistic Detection of Citation Hallucination in Long-Form RAG