Researchers increasingly build social simulations out of LLM agents that each play a person, such as an evacuating resident, a national delegate, or a forecaster, and then let them talk. To check that these agents behave faithfully, they usually ask another LLM to act as a judge: it reads each utterance and decides whether the utterance is supported by the scenario. This note studies a failure that only exists once agents talk to each other. Suppose one agent states an unsupported fact, for example "the Route 9 shelter is open." Its peers then repeat it. Because the judge reads the whole transcript, the claim now looks "supported in context." The fidelity check certifies a falsehood the simulation manufactured by itself. We call this repetition laundering.
We propose to measure laundering with planted claims, so that every claim's true origin is known. We test whether it grows with the number of repetitions and whether it depends on the persona of the agent who first said the claim. We also test whether it survives the obvious fix of using a judge from a different model family. Finally, we test a provenance-aware audit. It traces each claim back to the first time it appears in the transcript and judges the claim at that origin, rather than against the echo chamber that follows.
Core Claim
The claim is that support-in-context judging of multi-agent persona transcripts has a structural blind spot, separate from self-preference. The pass rate for an unsupported claim should rise with how often peers have repeated it before the judged utterance. It should rise further when the originating persona carries authority. And this should hold for self-judges and cross-family judges alike, because repetition turns into apparent evidence in the judge's own context window.
If this is right, the fix is not a better or different judge. The fix is to change what the judge is shown: each claim should be judged against its origin, while an agent who openly relays something a peer said should still count as faithful.
The idea sits on three pieces of prior work.
The motivating judge result is the third seed paper, When the Generator Grades Itself. On 107 frozen single responses, a 72B generator judging itself flagged 5.6% of outputs as unsupported, a matched-scale judge from another family flagged 53.3%, and a frontier judge flagged 92.5%, with disagreement running entirely one way. The author names live multi-turn persona transcripts as the open boundary. We cross that boundary, and we add the ground truth that the frozen design lacked. Without known labels, it is unclear whether 92.5% reflects strictness or accuracy.
The motivating amplification results come from two papers. Aligned Agents, Biased Swarm shows that downstream agents treat upstream outputs as established fact, and that a single planted premise polarizes fully connected agent networks. Van de Maele et al. show that passing processed beliefs between agents double-counts priors. Both measure bias inside the agent network. Neither asks whether the evaluator watching that network inherits the double-counting, which is our question.
The persona angle comes from two more papers. The Silent Judge finds that judges prefer content labelled with expert provenance over unknown provenance (an 18% shift in verdicts on ELI5 when the labels are swapped). It also finds that judges acknowledge such cues in their written rationales 0% of the time. Inside a simulation, a persona like "FEMA official" is exactly such a provenance cue, attached to a peer's claim. The 0% acknowledgment rate also constrains our design: we cannot detect laundering by reading the judge's rationales, so we must detect it by editing transcripts as counterfactuals.
Testbed and Ground Truth
We use three simulation settings, chosen so that the scenario facts are fixed and known.
The first is a hurricane-evacuation setting modelled on the HERO program of the second seed paper, The Effect Of Warning Message Length. Civilian personas from advantaged and disadvantaged populations receive real NWS and FEMA alert text, then discuss what to do. The alert text is the only legitimate source of facts.
The second is national-persona discussion on GlobalOpinionQA items, following Conformity, Confabulation, and Impersonation. There, each persona's onboarding answer and the Pew item define what that persona can legitimately assert.
The third is collaborative problem solving on GPQA, following From Single to Societal. It serves as an objective control in which unsupported claims are simply wrong.
In each setting, groups of 3 to 6 persona agents run in two topologies, a sequential chain and a fully connected group, borrowing the comparison from Aligned Agents, Biased Swarm. At a chosen turn we plant one unsupported claim in one agent's utterance. We vary the persona of that agent across three levels: authority (an official or expert), peer (a resident or delegate), and unlabelled. The simulation then continues naturally.
Every later utterance is decomposed into atomic claims. Each claim is linked to the planted claim by an entailment model, and two annotators verify a stratified sample. Each claim then receives one of three labels. It is supported if the scenario context supports it. It is a legitimate relay if it is attributed, as in "my neighbor says the shelter is open"; in a social simulation, relaying is faithful behavior. It is laundered if it asserts the planted content as fact without attribution.
Research Program
The central manipulation is a counterfactual edit of the transcript prefix, borrowing the counterbalancing logic of From Single to Societal. We hold the judged utterance fixed. In the transcript before it, we delete or duplicate peer repetitions of the planted claim, which gives a repetition count $k$ from 0 to 4. We also swap the originator's persona while keeping the words identical. Judges then score the identical target utterance under each edited prefix. Because the target never changes, any change in verdict is caused by the context.
We summarize the result with one regression:
$$\operatorname{logit} P(\text{pass}) = a + b\,\log(1+k) + c\cdot \text{authority} + d\cdot \text{selfjudge}.$$
Here $P(\text{pass})$ is the probability that a laundered claim is judged supported. The term $k$ counts prior unattributed repetitions. The terms authority and selfjudge are indicators for an authority-persona originator and for a judge from the generator's own family. The coefficient $b$ is the laundering slope. We use a logarithm because each extra repetition plausibly adds diminishing apparent evidence; we also report a free per-$k$ fit.
The constructive contribution is the provenance-aware audit. We build a claim lineage graph over the transcript. Each claim is judged only against the scenario context available to its originator at the turn it first appears. Every later occurrence inherits that verdict, unless it is explicitly attributed, in which case it is scored as a faithful relay. The judge model and rubric are unchanged. Only its input changes, and claim extraction plus linking costs a fixed number of extra calls per transcript.
Baselines
The naive reference is the default practice Sneiderman audits: the generator family judges each utterance against the full transcript. On the static proxy this flagged 5.6% where a matched cross-family judge flagged 53.3% (When the Generator Grades Itself). We expect its laundering slope to be the steepest.
The strongest single-judge competitor is Sneiderman's recommended fix, a matched-scale judge from a distinct family, together with the frontier cross-family judge. All judges use the same rubric, transcripts, and decoding.
The strongest multi-judge competitor is a stakeholder-persona panel with debate in the style of Multi-Agent-as-Judge. That panel reports Spearman correlation with humans of 0.47, against 0.31 for the single-judge method G-Eval, on StorySparkQA. We pair it with the independent-judges-plus-aggregator design that Judging with Many Minds found more bias-resistant than debate; that paper reports no numeric consistency scores. Both panels get a token budget matched to the provenance audit.
The key ablation is context stripping: each utterance is judged against the scenario context only, with no transcript. It isolates whether our gain comes from provenance tracing or merely from hiding peers.
Beating these baselines means two things at once on all three testbeds. The audit must cut the laundering slope $b$ to a value whose confidence interval includes zero. And its false-flag rate on legitimate relays must be no higher than the full-context judge's, which context stripping should fail to achieve.
Why not just…?
Why not just use a cross-family judge, as Sneiderman recommends? If laundering were a self-preference effect, this would suffice, and our contribution would reduce to confirming Sneiderman on live transcripts. That is a real possibility, and the regression settles it directly. A significant slope $b$ for the cross-family and frontier judges, with $d$ capturing only the self-judge's offset, means family change does not remove laundering. A slope of $b$ near zero for those judges means it does, and we would report that.
Why not just hide the transcript from the judge? This is the strongest threat, and it is the ablation above. Hiding peers removes laundering by construction. But it should also flag every faithful relay as unsupported, which penalizes exactly the behavior a social simulation is meant to produce. The comparison on relay false-flags decides whether lineage tracing earns its extra cost.
Why not just use a persona jury or debate panel? Panels average away independent errors. Laundering, however, comes from a shared context that every panelist reads. Judging with Many Minds also shows that debate amplifies judge biases after one round, and that corrective minority signals get diluted in aggregation. So we expect panels to show the same slope at several times the cost. The matched-budget comparison tests this.
Why not just read the judge's rationale for reliance on peers? The Silent Judge measured a 0% cue-acknowledgment rate. Rationale inspection is the wrong instrument, which is why the design relies on counterfactual prefix edits.
Decisive Checks
Three results decide the proposal.
The first is whether full-context judges show a positive laundering slope on the planted claims. A flat slope for all judges falsifies the core claim.
The second is whether the authority coefficient $c$ is positive in the evacuation setting. That would show persona provenance acting as a judge shortcut inside simulations, not only in attached metadata.
The third is whether the provenance-aware audit flattens the slope while keeping relay false-flags at the full-context level, where context stripping cannot.
As a secondary analysis, we compare the GPQA control with the subjective testbeds. This tells us whether laundering depends on the judge being unable to verify the claim independently, which bears on the reasoning-failure versus social-pressure question raised by Emergence of Biased Consensus in Multi-Agent LLM Debates.
Workshop authors to connect with
- Can LLMs Faithfully Enact Superforecaster Personas?: Andrew Robert Williams, Evan Jiang, Martin Weiss, Nasim Rahaman, Hugo Larochelle
- The Effect Of Warning Message Length on Language Model Simulated Response in Emergency Situations: Madeleine McDonald Gagné, Arnav Jhala
- When the Generator Grades Itself: A Self-Judge Under-Flags Unsupported Outputs at Matched Scale in LLM Social Simulation: Rob Sneiderman
References
- When the Generator Grades Itself: A Self-Judge Under-Flags Unsupported Outputs at Matched Scale in LLM Social Simulation
- Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent Systems
- Belief Sharing: A Blessing or a Curse
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- The Effect Of Warning Message Length on Language Model Simulated Response in Emergency Situations
- Conformity, Confabulation, and Impersonation: Persona Inconstancy in Multi-Agent LLM Collaboration
- From Single to Societal: Analyzing Persona-Induced Bias in Multi-Agent Interactions
- Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
- Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplification and Resistance in Multi-Agent Based LLM-as-Judge
- Emergence of Biased Consensus in Multi-Agent LLM Debates
- Can LLMs Faithfully Enact Superforecaster Personas?