TL;DR
Large Language Models (LLMs) are easily tricked into believing misinformation if it is repeated across multiple news outlets, even when the reports are verbatim copies of a single source. This paper demonstrates that "copy attacks" significantly inflate an LLM's belief in false claims and proposes an "entity-novelty" test to identify reworded duplicates that standard similarity detectors miss.
What Problem Does This Paper Address?
In the real world, news is often "recycled." A single wire story from Reuters or the AP might be reprinted by dozens of local outlets. While humans (and journalists) recognize these as a single source of information, the authors ask: do LLMs?
This is a critical question for continual learners—models that update their knowledge base by reading the news—and LLM judges used for fact-checking. If a model treats ten reprints as ten independent confirmations, it becomes highly vulnerable to "churnalism" and coordinated misinformation. The paper explores how common this duplication is, how it fools current models, and how we might defend against it.
Methods
The researchers employed a multi-pronged approach:
- Duplication Analysis: They used MinHash and Locality-Sensitive Hashing (LSH) on 200,000 news articles to measure real-world duplication and developed a taxonomy of dependence (e.g., wire copy, same-owner sharing, syndication).
- Controlled Judge Experiments: They tested four models (Llama 3.2 3B, phi4 14B, Gemini 2.5 Flash, and Gemini 2.5 Pro) by presenting them with claims backed by varying numbers of reports, sometimes including explicit "provenance notes" stating the reports were duplicates.
- Simulated Continual Learning: They built a stream of news reports where a "student" model (Qwen 2.5) updated its beliefs based on the verdicts of an LLM judge.
- Entity-Novelty Test: To catch LLM-generated rewrites that bypass standard text-similarity tools, they developed a test that measures the share of "new" named entities (people, places, organizations) in a report compared to earlier ones.
Datasets
- CC-News: A sample of 200,000 English articles (2017–2019) used to establish baseline duplication rates.
- CounterFact: Used to generate "known-fact lies" (counterfactuals) to test if models would override their internal knowledge when faced with multiple "confirming" reports.
- Custom Labeled Sets: The authors manually labeled 300 pairs of articles to validate their dependence taxonomy and 100 pairs of independent reports of the same event to test detector precision.
Key Insights and Ideas
- Repetition is a "Truth" Signal: For most models, seeing a claim three times instead of once makes it feel significantly more "true," even if the text is identical.
- Provenance Blindness: Smaller models like Llama 3.2 and phi4 largely ignored explicit warnings that reports were reprints. They prioritized the count of reports over the source of information.
- The "Flash" vs. "Pro" Divide: Gemini 2.5 Flash was highly "gullible," believing claims based on a single unverified report. In contrast, Gemini 2.5 Pro was the only model to successfully use provenance tables to discount duplicates.
- Lexical Similarity is Not Enough: Standard Jaccard similarity (looking at shared words) catches verbatim copies but is easily evaded by LLM rewrites.
- Entities as Information Proxies: The "aha moment" of the paper is that while an LLM can change every word in a story, it usually keeps the names and numbers. Measuring "entity novelty" is a robust way to identify if a report actually adds new information or is just a rewrite.
Key Results
The scale of the "copy problem" is massive: 18.9% of the CC-News sample consisted of near-copies.
In the judge experiments, the impact of duplication was stark. For Llama 3.2 3B, the acceptance of lies jumped from 0.06 (with one report) to 0.71 (with three copies).
The proposed entity-novelty test proved highly effective, separating LLM rewrites from independent reports with an AUC of 0.98, vastly outperforming standard text-similarity detectors which failed to catch rewrites entirely. By filtering out these duplicates in a simulated learning stream, the researchers reduced the acceptance of reworded lies from 0.67 to 0.20.
Limitations and Future Work
- Collusion: The current methods rely on detecting textual or entity-based similarity. If two outlets independently coordinate to spread the same lie without sharing a source text, these detectors will fail.
- Natural Rewrites: Professional journalists often add enough original reporting (new quotes or context) to bypass the entity-novelty test, even if the core claim is a derivative.
- Real-world Validation: The study uses simulated streams and invented claims to isolate variables; the authors note that the pipeline needs end-to-end validation on real, messy news cycles.
- The "Wait and See" Strategy: Future work should explore giving LLM judges the option to "wait" for independent evidence rather than forcing a True/False verdict on the first report they see.
Summary updated