When a language model answers a question about Basque social norms or Japanese workplace hierarchy and is handed a few retrieved documents, its answers change. The usual reading is that the model used the evidence. An equally plausible reading is that the model simply reacted to having extra text in its prompt. Prior work has already caught this second effect with persona prompts: models shift their answers just as much when told an irrelevant "favorite planet" as when told a country of origin. We propose to run the same kind of control for retrieval. Every real evidence document is paired with a placebo document, a passage in the same language, of the same length and register, that carries no relevant cultural content. The part of a model's behavior change that survives this subtraction is the part that came from grounding.
The opportunity matters most outside English. Retrieval in low-resource languages returns noisier, sparser, and often English-language pages. The methods that make models ignore irrelevant passages have been validated almost entirely on English Wikipedia question answering. If placebo documents move Basque or Japanese answers as much as real evidence does, then retrieval-augmented "cultural adaptation" is largely prompt sensitivity under another name. We would also have a cheap correction: a decoding rule that subtracts the placebo-induced shift.
Core Claim
We claim that measured cultural grounding from retrieved context is confounded with a generic context-presence effect. We further claim this confound is larger in low-resource languages, and larger when the evidence is in a different language from the question. Three observations motivate the claim.
The first is about persona prompts. Cultural Conditioning or Placebo? showed that non-cultural cues flip Llama 3's MMLU answers up to 15 out of 50 times, about as often as cultural cues do. That study used no retrieval at all.
The second is about retrieval and bias. Information Retrieval Induced Safety Degradation in AI Agents showed that giving LLaMA 3.2 web access drops refusal on biased BBQ prompts from 0.80 to 0.46, and that safety instructions in the system prompt do not stop this. That study was explicitly English-only and did not separate content effects from context-presence effects.
The third is that distraction is well documented in English open-domain question answering. Zero-shot readers collapse as more passages are added (Improving Zero-shot Reader by Reducing Distractions), and summarizers lose 7.4 ROUGE points when irrelevant documents are added (Open Domain Multi-document Summarization).
Nobody has crossed these findings: a matched placebo control applied to retrieved cultural evidence, across languages. What would be new is that crossed measurement. A second, smaller contribution is a mitigation built directly from the control.
We define the placebo-adjusted grounding effect for a model on a benchmark as
$$G = \Pr(\text{correct} \mid E_{\text{rel}}) - \Pr(\text{correct} \mid E_{\text{plc}}),$$
where $E_{\text{rel}}$ is relevant cultural evidence and $E_{\text{plc}}$ is a matched placebo of identical length, language, and register.
We set this against the naive grounding effect, which compares $E_{\text{rel}}$ to no context. The gap between the naive and adjusted effects is the context-presence artifact. Our hypothesis is that this gap is near zero for English questions with English evidence. We expect it to be a substantial fraction of the naive effect for Basque, and largest when the question is Basque and the evidence is English.
Research Program
The core artifact is an evidence layer attached to existing cultural benchmarks with closed answer sets. We use four benchmarks. BasqBBQ supplies native Basque bias QA together with its translation controls, BBQ2eu and BasqBBQ2en. SOBACO, from Bias Mitigation or Cultural Commonsense?, supplies Japanese hierarchy and honorific commonsense, with JCommonsenseQA as a culture-neutral control. CALI and EtiCor, used by the placebo paper, supply culturally sensitive judgments. Closed answer sets matter because they let us score with constrained decoding. That removes the parser-artifact explanation for answer variance.
For each item we prepare five evidence conditions. The no-context condition is the baseline. The relevant-cultural condition is a passage that states the operative norm, drawn from native-language encyclopedic or etiquette sources and checked by a native annotator. The matched placebo is an on-language passage of equal length and register on an unrelated topic. The wrong-culture condition states the Anglophone version of the norm; for example, a passage that treats seniority-based address as optional. The stereotype-laden distractor is topically adjacent text that contains group stereotypes; this mirrors what open-web retrieval returned in the safety-degradation study.
Each evidence condition appears in both the question's language and English. Each item is also posed under twelve semantically equivalent templates, following ProSA, at 0-shot and 1-shot.
The analysis extends the variance decomposition of Brittlebench. Brittlebench splits performance variance into data difficulty and presentation brittleness. We add two more components: context presence, estimated from the placebo-versus-none contrast, and context content, estimated from the relevant-versus-placebo contrast. We estimate all four per language and per model scale. For BasqBBQ we additionally test the reported scale effect, in which larger models amplify stereotypes in ambiguous contexts. Specifically, we ask whether the stereotype-laden distractor amplifies that effect more in Basque than in English.
The mitigation, placebo-contrastive decoding, follows directly from the measurement. For each answer option $y$ we compute
$$s(y) = \log p(y \mid E_{\text{rel}}) - \alpha\,[\log p(y \mid E_{\text{plc}}) - \log p(y \mid \varnothing)],$$
where $p$ is the model's option probability, $\varnothing$ is the empty context, and $\alpha \ge 0$ is tuned on a held-out split. The rule removes whatever shift a content-free passage of the same language and shape would have induced. It runs on frozen open-weight models and costs two extra forward passes per item. Its key ingredient is that the placebo is matched to the evidence.
Baselines
The naive reference point is the no-retrieval model under a single fixed template. A second naive reference is plain retrieval-augmented prompting, with evidence simply prepended. Every other method must beat plain retrieval on placebo-adjusted grounding $G$ and on ambiguous-context bias score, not only on raw accuracy.
Prompted Chain-of-Note is the strongest existing method for ignoring irrelevant context. It asks the model to write a note on each passage, labeling it relevant, irrelevant-but-helpful, or irrelevant, before answering. Chain-of-Note reports +7.9 EM on average across NQ, TriviaQA and WebQ when all retrieved documents are irrelevant, and +10.5 correct rejection on RealTimeQA. That paper fine-tunes its models, whereas we can only prompt open-weight models. We therefore report our version as a prompted lower bound and state this plainly.
Distraction-aware Answer Selection is the second retrieval-robustness baseline. It combines an "unanswerable" instruction with reranking by query-generation likelihood. Improving Zero-shot Reader by Reducing Distractions reports an average 64% EM increase over naive zero-shot readers on English open-domain QA. Its question-likelihood score may transfer poorly to Basque, and that is exactly what makes it a useful test.
For the selective-answering view we compare against SENSEL, which abstains when predictions are unstable across prompt paraphrases. On the Relation between Sensitivity and Accuracy reports +4.1 AUC over a max-probability baseline on GPT-J-6B. Our variant scores instability across placebo documents instead of across paraphrases.
All methods use the same models, the same evidence, and the same twelve templates, and are scored with constrained option scoring. Any method with hyperparameters gets the same held-out tuning split and the same number of tuning trials.
The ablation that isolates our key ingredient replaces the matched placebo with an unmatched one, such as an English passage or a passage of different length. It also runs the subtraction against the empty context alone. If unmatched placebos work just as well, the matching claim fails.
We count a win as follows. Placebo-contrastive decoding must improve Basque and Japanese accuracy under the relevant-evidence condition. It must lower the BasqBBQ ambiguous-context bias score under the stereotype-laden distractor. It must not lose more than one point on the English controls. It must outperform prompted Chain-of-Note on at least one of the first two criteria at a fraction of the cost. For none of these benchmarks under retrieval did we find a reported result on Lacuna, so all numbers will be new.
Why not just…?
Why not just use an LLM judge, since prompt sensitivity is an evaluation artifact? Flaw or Artifact? shows that template variance on ARC-Challenge falls from a standard deviation of 0.28 to 0.005 once a judge model replaces string matching. That argument targets answer parsing. All of our tasks have closed answer sets scored by constrained option probability, so there is nothing to mis-parse. Any variance that remains belongs to the model. We will still run a judge on a free-generation subset. This tests whether a mostly-English judge reports false stability on Basque and Japanese outputs relative to native-annotator labels.
Why not just tell the model to use only relevant evidence, or to stay unbiased? The literature already argues against this from both sides. Safety instructions are overridden once retrieved context is present (Information Retrieval Induced Safety Degradation). When models do obey debiasing instructions, Japanese cultural-commonsense accuracy collapses by up to 75% (Bias Mitigation or Cultural Commonsense?). We include an instruction-only condition anyway, because the combination of instruction and retrieval in non-English settings is untested.
Why not just use Chain-of-Note or another relevance filter, which already handles irrelevant passages? This is the real threat. If prompted note-taking pushes the placebo-induced shift to near zero in Basque, our mitigation is unnecessary. The measurement would still stand, because it shows how large the problem is before filtering. The experiments settle this directly: we report the context-presence variance component with and without Chain-of-Note, per language. One difference remains even if Chain-of-Note does well. Our placebos are fluent, on-language, and topically plausible, whereas filtering methods are tested against obviously off-topic passages. Matched placebos are a harder case for any filter.
Why not just measure paraphrase brittleness, as Brittlebench does? Paraphrase variance cannot tell a model that read the evidence from one that reacted to extra tokens. Both look identical under template perturbation. The placebo-versus-relevant contrast is the only part of the design that separates these two explanations.
Proposed Experiments
The models are Llama-3.1 at 8B and 70B, Latxa-70B as a Basque-adapted model, and Qwen and Gemma models of comparable sizes. This spans the scale axis on which BasqBBQ reports stereotype amplification.
The first decisive check is the gap between naive and adjusted grounding, broken down by question language and evidence language. Our claim fails if this gap is as small in Basque and Japanese as it is in English.
The second check asks whether matched placebos move answers more than template paraphrases do. A positive answer would establish context presence as a variance source separate from what Brittlebench measures.
The third check asks whether English-language evidence for Basque questions yields a smaller adjusted grounding effect than Basque-language evidence carrying the same content. That would be a geographic-bias failure created by the retrieval pipeline itself.
The fourth check tests whether the 1-shot anchoring that ProSA reports survives under retrieved noise, or whether the 1-shot demonstration becomes one more distractor. This would arbitrate the ProSA–Brittlebench disagreement in a multilingual retrieval setting.
The last check is the placebo-matching ablation. If unmatched placebos subtract as effectively as matched ones, the proposal reduces to ordinary contrastive decoding, and we will report it that way.
References
- Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Prompting
- Information Retrieval Induced Safety Degradation in AI Agents
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval
- BasqBBQ: A QA Benchmark for Assessing Social Biases in LLMs for Basque, a Low-Resource Language
- Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Brittlebench: Quantifying LLM robustness via prompt sensitivity
- Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs