When the Reader Grades the Retriever: Cultural Bias in Model-Feedback Retrieval

A common way to train the search component of a retrieval-augmented system is to let the downstream model grade it. Retrieval-augmented generation means a model answers using documents fetched by a separate retriever. In the grading setup, a document counts as good if showing it to the answering model (the reader) makes the correct answer more likely, and the retriever is trained toward those documents. This removes the need for human relevance labels, and it works well on English Wikipedia questions. Our hypothesis is that it fails in a predictable way on culturally specific questions. A reader that already confuses Safavid Persian art with Mughal Indian art, or folds a regional dish into a better-known neighbor, will rate documents about the familiar neighbor as "useful," because those documents agree with what it already believes. The retriever then learns to fetch evidence that confirms the reader's blind spots. The feedback loop that was meant to close the cultural gap could widen it.

We propose to measure this effect directly and to fix it with a small change to the grading signal. A document should earn credit only if it does two things. It should raise the reader's probability of the correct answer beyond what the reader believed with no document. It should also lower the probability of the specific wrong answer the reader was most tempted by. In two sentences: model-graded retrievers inherit the grader's cultural confusions, because likelihood rewards confirmation. We correct this by scoring documents on how much they move the reader away from its own most likely mistake, not on how much they raise its confidence.

Core Claim

Several lines of evidence point to this failure mode, though none tests it directly. ReSCORE trains multi-hop retrievers from an LLM's answer-consistency score $P(a|q,d)$. It reports a "false positive trap": supervising on consistency alone teaches the retriever to prefer distractors that contain the answer string without supporting it. VULCA-Bench documents "cultural conflation" between adjacent traditions. It also documents an "illusion of expertise," in which models use sophisticated cultural vocabulary that is not anchored to the image. M4-RAG finds that retrieved cultural text helps small vision-language models (Gemma3 4B gains 5.74% on CVQA) but hurts larger ones (Gemma3 27B loses 1.75%). It also reports a 20–30% gap between the best retrieval-augmented systems and an oracle given the right context.

Put together, these suggest a specific mechanism. In cultural retrieval, the most dangerous distractors are documents about a neighboring culture, because they share religion, geography, or vocabulary with the target. A reader with a skewed prior assigns them high likelihood. Likelihood-based feedback, as in REPLUG (which distills the reader's document-conditioned answer likelihood into the retriever through a KL-divergence objective), therefore rewards them. Nothing in the literature compiled here measures this. The new contribution is the diagnosis plus a corrected utility. For a question $q$ with correct answer $a^*$, let $\tilde a$ be the reader's highest-scoring wrong answer with no context. For multiple-choice data such as CVQA, $\tilde a$ is the most-preferred wrong option. For open-ended answers, it is the reader's top wrong sample. We score a document $d$ as

$$u(d) = \big[\log P(a^*|q,d) - \log P(a^*|q)\big] - \lambda \big[\log P(\tilde a|q,d) - \log P(\tilde a|q)\big],$$

where $P$ is the frozen reader's probability and $\lambda \ge 0$ weights the penalty on reinforcing the tempting mistake. The first bracket removes credit for documents that merely restate what the reader already knew. This targets the inverse-scaling finding, since a large reader gains nothing from confirmation. The second bracket removes credit for neighbor-culture documents that raise the probability of every culturally adjacent answer at once. The retriever is trained, as in REPLUG, to match $\mathrm{softmax}(u(d)/\tau)$ over its top candidates, where $\tau$ is a temperature. The construction needs no new annotation beyond the gold answer the feedback literature already uses.

Research Program

The work proceeds in three stages. Each stage can stop the project if its premise fails.

The first stage is a diagnostic. On RAVENEA, which pairs culture-centric visual questions from eight countries with human-ranked Wikipedia documents, we build matched document pairs for each question. One document is gold-culture; the other is a same-category document from the most-confused neighboring country, chosen by the reader's no-context errors. We then compute standard REPLUG utility, $\log P(a^*|q,d)$, for both documents. The readers span sizes and families: LLaVA-1.5-7B, Gemma3 4B, and Gemma3 27B. The question is whether neighbor documents receive higher utility than gold documents more often for countries where the reader's no-context accuracy is low. If this correlation is absent, the core premise is wrong. The project then becomes a negative-result paper, and we do not proceed to stage two as framed.

The second stage trains retrievers. Starting from the same initialization, we train with standard likelihood utility, with prior-corrected utility (the first bracket only), and with the full contrastive utility. The training data are the RAVENEA and CVQA training splits, and the retriever is evaluated with each reader frozen. The analysis tracks average accuracy, worst-country accuracy, and the share of retrieved top-1 documents whose culture matches the question.

The third stage tests the multi-hop regime, which is where we expect the effect to compound. In label-free iterative training such as ReSCORE, intermediate hops are supervised using the model's own intermediate answers, so a conflated first hop teaches the retriever to keep fetching the wrong culture. We build a small two-hop variant of RAVENEA. Hop one identifies the cultural entity in the image. Hop two asks a linked fact from that entity's document, such as the region or period, which must then be retrieved. We apply the contrastive utility at each hop, with $\tilde a$ set to the reader's tempting wrong intermediate entity. This extension is constructed by us, so no reported results exist on it. We present it as a stress test, not a headline benchmark.

Baselines

The naive reference point is the reader with no retrieval, and the same reader with an off-the-shelf frozen retriever. M4-RAG uses frozen retrievers in this way and supplies the published direction of effect: gains for small readers, losses for 27B readers.

The strongest existing retriever on this data is Ravenea-CLIP from RAVENEA. It is trained with culture-aware contrastive learning on human relevance rankings and reports a Mean Reciprocal Rank of 82.17. That paper also reports that culture-aware retrieval improves vision-language models by about 6% on culture-centric visual question answering. This is the competitor to beat, because it attacks the cultural gap with human supervision instead of model feedback. If human relevance already solves the problem, the proposal is unnecessary. We run it with its released training recipe on the same splits. Our method is initialized from the same CLIP backbone and receives an equal number of retriever update steps and the same hyperparameter search budget. Beating it means higher downstream answer accuracy and higher worst-country accuracy with the same reader, not higher MRR. Our claim is precisely that MRR against human ranks is the wrong target.

The strongest model-feedback baseline is REPLUG-style likelihood distillation, re-implemented on the same retriever, readers, and candidate pool. No reported result found on Lacuna for REPLUG on RAVENEA or CVQA. Its published numbers, such as 45.5 on few-shot Natural Questions with Codex, are text-only and not comparable. For the multi-hop stage, the reference is ReSCORE's relevance-plus-consistency objective. We cite its reported 23.4 exact match on MuSiQue and 50.0 on 2WikiMultiHopQA as evidence that the objective is strong on English text. We re-run it on our two-hop cultural variant under matched compute.

Two ablations isolate the key ingredient. The first, prior-corrected utility with $\lambda=0$, tests whether subtracting the no-context baseline is enough. The second keeps the contrastive term but replaces $\tilde a$ with a random wrong option. This tests whether the reader's own tempting mistake matters, or whether any negative answer would do.

Why not just…?

Why not just keep human relevance labels, as Ravenea-CLIP does? Human relevance and reader utility come apart. M4-RAG's 20–30% oracle gap and its 27B regression both occur with retrieved text that is topically relevant. However, Ravenea-CLIP is a real threat on RAVENEA itself, where its labels are dense and high quality. Our case for model feedback rests on settings without such labels: CVQA's 30 countries and the two-hop variant. If Ravenea-CLIP matches our method on worst-country accuracy there, we will report that model feedback adds nothing.

Why not just make the reader ignore bad documents, with distraction-aware answer selection or Ext2Gen? Distraction-aware answer selection reweights answers by query likelihood and adds an abstention option. It reports large exact-match gains on open-domain QA as documents grow from 5 to 100. Ext2Gen trains the generator to extract relevant sentences first, raising Llama-3.1-8B-Instruct from 0.473 to 0.557 under heavy noise. Both assume the reader can recognize a distractor. A neighbor-culture document is not obviously irrelevant to a reader that shares the confusion, so reader-side filtering should keep it. We test this by applying distraction-aware selection on top of the standard-feedback retriever and measuring the conflation rate of errors. If that rate falls as much as it does under our retriever, this alternative wins.

Why not just subtract the no-context prior and skip the contrastive term? This is the most plausible simpler fix, and it is our first ablation. Prior correction alone should handle the inverse-scaling effect. It does not distinguish a gold document from a neighbor document that raises both $a^*$ and $\tilde a$. We expect the two to tie on high-resource countries and diverge on the most-confused pairs. If they tie everywhere, we drop the contrastive term and report the simpler method.

Why not just upsample under-represented countries during retriever training? Reweighting changes how often each country is seen. It does not change which document the reader's grade favors within a country, and that within-country choice is the hypothesized failure. We run reweighted standard feedback as an additional arm.

Proposed Experiments and Decisive Checks

The evaluation uses RAVENEA culture-centric visual question answering and CVQA multiple choice, both in English and in local-language prompts. We include the local-language prompts because CVQA reports LLaVA-1.5-7B falling from 49.6% to 35.5%, and the conflation effect may be strongest there. The two-hop RAVENEA variant is the stress test. MuSiQue serves as a check that the contrastive utility does not hurt on English text, against ReSCORE's 23.4 exact match.

Metrics are answer accuracy, worst-country accuracy, conflation rate (the fraction of errors that equal $\tilde a$ or a same-region option), culture-match rate of the top retrieved document, and the retrieval gain separately for 4B and 27B readers.

The decisive check comes in two parts. Stage one must show that standard likelihood feedback prefers neighbor-culture documents more often for low-accuracy countries. Stage two must show that the contrastive retriever cuts the conflation rate and lifts worst-country accuracy above both Ravenea-CLIP and REPLUG-style feedback, while erasing M4-RAG's negative retrieval gain for the 27B reader. If the bias exists but prior correction alone removes it, the contribution narrows to the diagnosis and the simpler fix. That would still be a useful result for anyone training retrievers from model feedback outside English Wikipedia.