Language models that answer questions by first fetching documents, known as retrieval-augmented generation (RAG), are now routinely defended against misinformation by trusting some sources more than others. Current defenses either down-weight low-credibility documents inside the model or vote across sources by their estimated reliability. These defenses treat "credible" as meaning "factually correct." A source can be accurate on every checkable claim and still frame a social group stereotypically. If its trust score boosts everything it says, the framing is boosted too.
We propose to test whether credibility weighting in RAG amplifies societal stereotypes in this way, and to fix it by attaching trust only to the falsifiable claims in a document rather than to the source as a whole. The study has two parts. The first is a controlled testbed that varies a source's trust label, its factual correctness, and its stereotyped framing independently. The second is a claim-level variant of an existing attention-based defense, compared head to head against the source-level version using the same credibility estimates.
Core Claim
Source-level credibility weighting carries two separate routes for bias. The first is the evaluator: when a language model scores credibility, it brings in prestige halo effects. Investigating Subtler Biases in LLMs shows that GPT-4, PaLM-2, Llama-2 and Mistral ascribe reliability and creativity to elite-university graduates and rich nations even with education held fixed. The second is selection: whoever ends up trusted, their framing gets amplified along with their facts. A real-world hint of this second route comes from News Source Credibility Assessment: A Reddit Case Study. That work builds a credibility classifier (a text encoder plus a graph network over Reddit posts) and reports that credibility scores correlate with political lean.
Removing source identity before scoring ("blinding") can close the first route but not the second. The second route closes only if trust stops spreading from a document's checkable claims to its descriptive framing of people. The claim is falsifiable. If blinding alone removes the bias increase, or if claim-level weighting costs substantial factual accuracy, the proposal fails.
Research Program
Methods we build on. The intervention mechanism comes from CrAM. CrAM finds the attention heads that most influence answers, using causal tracing (measuring how much an output changes when a component's activation is patched). At inference time it scales down the attention those heads pay to tokens from low-credibility documents.
The decomposition comes from ReliableRAG. ReliableRAG splits retrieved text into subject–predicate–object triples and scores each triple's credibility.
A label-free reliability estimator comes from RA-RAG. RA-RAG estimates per-source reliability by cross-checking sources against each other and then uses weighted voting.
All three target factual misinformation on NQ, TriviaQA and HotPotQA. None of them measures demographic bias, which is the gap this proposal fills.
Proposed method: claim-scoped CrAM. Each retrieved passage is segmented into two kinds of span. Claim spans contain triples that can be checked against other sources. Framing spans hold everything else, including descriptors of people and groups. The credibility weight, whether boosting or suppressing, is applied to tokens in claim spans. Framing spans receive the neutral weight that the unweighted model would give them.
We do not try to classify framing as "biased." That would import a debiasing classifier and its own biases. The rule is simply that trust earned on facts does not transfer to descriptions.
Testbed. The testbed is built on BBQ and StereoSet, the benchmarks Bias Amplification in RAG used to show retrieval can amplify age and gender bias. For each item we write retrievable passages in a full factorial design over three factors:
- The passage contains the answer-relevant fact, or does not.
- The passage carries stereotype-consistent framing of the group, stereotype-inconsistent framing, or neutral framing.
- The passage is attributed to a source of high or low credibility.
Source names come from two places. Outlet names are taken from the 223 labeled news sources in the Reddit credibility dataset, which carry Ad Fontes and Media-Bias-Fact-Check labels. Prestige names are drawn from the university and nationality categories in the Kamruzzaman et al. benchmark.
The design lets us estimate, by regression on the factors, how much a stereotyped answer depends on framing given trust level. It also lets us compare credibility from an LLM judge, credibility from RA-RAG consensus, and oracle credibility.
The adversarial setting of the BRRA paper is reframed here as an organic one. No attacker injects documents; every document is factually correct.
Baselines
The do-nothing reference is vanilla RAG over the same retrieved passages, with no credibility signal. Every bias and accuracy number is reported relative to it, so "amplification" has a fixed meaning: a higher stereotype-consistent answer rate than vanilla RAG on ambiguous BBQ contexts.
The strongest existing method is source-level CrAM. It is run on the same open models that CrAM itself used (Qwen1.5-7B, Llama2-13B, Llama3-8B). Head selection follows CrAM's calibration recipe, using the same calibration set and head count for both CrAM variants. The two variants use identical credibility scores, so the only difference is how far the weight reaches within a document.
On factuality, CrAM reports TriviaQA exact-match gains over prompt-based credibility of +25.5% (Qwen1.5-7B), +31.9% (Llama2-13B) and +10.9% (Llama3-8B), under the 4-real/1-fake setting. We first reproduce that setting to confirm our implementation. We found no reported result on Lacuna for CrAM, ReliableRAG or RA-RAG on BBQ or StereoSet; producing those numbers is part of the contribution.
ReliableRAG is a second strong competitor because it already reasons over triples. On HotPotQA under misinformation injection it reports 55.10% exact match against 20.10% for vanilla RAG. It is run with its α = 0.4 balance and the same retriever. The question it answers is whether triple-level filtering already discards framing as a side effect.
RA-RAG weighted voting is run with consensus-estimated reliability. It represents a credibility signal that does not depend on any judge's sense of prestige.
Blinded source-level CrAM scores credibility with outlet and institution names masked. It isolates the evaluator route.
Generation-stage fairness guidance on top of source-level CrAM comes from the BRRA defense. It is the cheapest generic fix.
Every baseline gets the same retrieved passages, the same models, and a tuning budget matched in validation queries.
Claim-scoped CrAM beats source-level CrAM if two things hold on ambiguous BBQ contexts. It must cut the bias increase over vanilla RAG by at least half. It must also keep TriviaQA exact match within two points of source-level CrAM. Both are pre-registered thresholds.
Why not just…?
Why not just blind the credibility scorer to source identity? This is the most serious alternative, and the testbed is designed around it. If amplification flows only through the evaluator's prestige halo, blinded CrAM should match claim-scoped CrAM, and we will report that outcome as the finding. Our hypothesis is that blinding fails under oracle credibility. With a perfectly prestige-free trust label, source-level weighting should still amplify the stereotyped framing of trusted sources. The oracle condition in the factorial design separates the two routes directly.
Why not just add a fairness instruction at generation time? Uncovering Stereotypes in Large Language Models shows that prompt-level guardrails erode as tasks get harder: ChatGPT's refusal rate falls from 32.8% to 29.4% across complexity tiers. CrAM shows that models follow credibility cues in prompts poorly. We still run this baseline. We also evaluate on multi-hop variants of the testbed passages, where this prior work predicts the instruction will weaken.
Why not just use consensus reliability, which never sees prestige? Consensus measures agreement. If most trusted outlets share a framing, agreement rewards that framing. RA-RAG validated its scores against PolitiFact truthfulness (Pearson correlation 0.991), which says nothing about representation. Running RA-RAG in the factorial design tests this directly.
Why not just drop low-credibility documents outright? CrAM and ReliableRAG both report that hard exclusion discards useful context. Exclusion also does nothing about framing in documents that are kept. We include it only as a sanity check.
Proposed Experiments and Decisive Checks
The first check is existence. Does source-level CrAM, under LLM-judged credibility, raise the stereotype-consistent answer rate on ambiguous BBQ items and StereoSet above vanilla RAG? We also check whether it raises institutional and nationality halo correlations (Kendall's τ, as used by Kamruzzaman et al.) when retrieved passages mention institutions. A null result here ends the project cheaply.
The second check is decomposition. In the factorial regressions, does the interaction between stereotyped framing and high trust stay significant under oracle and blinded credibility? If yes, the selection route is real. If no, blinding suffices.
The third check is the head-to-head. We report claim-scoped against source-level CrAM on BBQ bias, BBQ disambiguated accuracy, and misinformation-injected TriviaQA and NQ exact match.
The fourth check uses natural rather than written passages. We retrieve real articles linked from the Reddit dataset's high- and low-credibility outlets about adolescents and gendered emotion. The probes are built from the framings documented by Representation Bias of Adolescents in AI and Angry Men, Sad Women. This tests whether the effect appears outside written passages.
Finally, we run a five-round retrieval–generation loop. In each round the model's answers are written back into the corpus with the source's credibility label. This tests whether the amplification compounds, as the iterated-learning analysis of Bias Amplification in Language Model Evolution would predict when a weighting rule concentrates mass on one mode.
References
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models
- News Source Credibility Assessment: A Reddit Case Study
- CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAG
- ReliableRAG: Combating Misinformation in Retrieval-Augmented Generation via Reliability-Guided Reasoning Chains
- retrieval augmented generation with estimation of source reliability
- Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- Representation Bias of Adolescents in AI: A Bilingual, Bicultural Study
- Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attribution
- Bias Amplification in Language Model Evolution: An Iterated Learning Perspective