Rumor detectors score above 90% on standard benchmarks, then fall apart on news events they have never seen. One reason is structural. In datasets such as Twitter15, Twitter16 and PHEME, each real-world event tends to carry mostly one veracity label, so a model can tell true from false largely by recognizing which event a post is about. When posts are split so that training and test events never overlap, accuracy drops by more than 40% (Probing Spurious Correlations in Popular Event-Based Rumor Detection Benchmarks). We propose to remove that shortcut at the source. A large language model (LLM) writes new training posts about each event in the style and claim structure of real posts that carry a different label and come from different events. After this rewriting, knowing the event tells the classifier nothing about the label.
The piece that makes this more than paraphrasing is choosing which real post to borrow structure from. We call that post the "donor." We retrieve donors only from other events and other labels, and we rank them by similarity after masking out names, places and hashtags. The goal is a donor whose claim type fits the target event, such as a casualty count, an attributed quote, or a "breaking: X has died" announcement. The rewritten post should then be coherent rather than a mechanical collage. Existing LLM augmentation for fake news does the opposite. RumorLLM rebuilds rumors around TF-IDF keywords taken from the same training events and is evaluated only on random, event-mixed splits. That multiplies exactly the vocabulary a detector should not rely on.
Core Claim
Event-specific overfitting in text-based rumor detection is mainly a problem of label-skewed training data. It can be reduced more effectively by generating event-balanced counterfactual posts than by erasing event information from the model's internal representations. Let $E$ be the event a post belongs to and $Y$ its veracity label. Our augmentation aims to drive the statistical dependence between them toward zero in the training set. We measure that dependence with their mutual information $I(E;Y)$, the amount that knowing one variable reveals about the other. Unlike representation-level methods, the augmentation keeps each post's real content intact.
We make three predictions. First, a plain text classifier trained on transplant-augmented data should close most of the gap to graph-based unseen-event methods under event-separated splits. Second, entity-masked, cross-event donor retrieval should beat both random donors and ordinary sentence-embedding retrieval. Third, the gain should come with a measurable drop in how well event identity can be read off the classifier's features.
Relation to Prior Work
The motivating evidence comes from benchmark audits. Wu and Hooi (Probing Spurious Correlations) showed the accuracy collapse under event-separated splits. Jiang et al. found that masking event keywords degrades standard models. Guo et al. found the analogous entity memorization in multimodal detectors.
Existing fixes act on the model rather than the data:
- MetaDetector trains an event discriminator adversarially, and adds importance weights to avoid negative transfer from event-idiosyncratic posts.
- FADE perturbs graph embeddings toward the decision boundary and subtracts the output of an event-only predictor.
- FNDCD down-weights samples that a causal model flags as environment-biased.
All three either erase or down-weight event-bound signal. None adds counterexamples that the data never contained. That limit matters most when an event has almost no posts of the minority label: no amount of reweighting creates a false rumor about an event that has only true ones.
Our method borrows from two lines of work. The first is counterfactual augmentation. Iterative Counterfactual Data Augmentation swaps label-bearing segments across classes to break spurious correlations in sentiment data. We transfer the idea to swapping claim structure across events. The second is structure-aware demonstration retrieval. GPT-RE showed that surface-similarity retrieval pulls relationally irrelevant examples and fixed this with entity-focused and task-specific representations. We apply that insight to donor selection.
PROPANEWS is the closest generation-side work. It injects propaganda techniques into authentic articles, but it never breaks the event–label link and never tests event-separated splits.
The style-robustness results of Przybyła and the stance/topic disentanglement of Dougrez-Lewis et al. support the premise that event-invariant signal exists in form and stance.
One constraining result must be faced directly. In Bangla fake news augmentation, similarity-based subsampling with Sentence-BERT (SBERT, a sentence-embedding model) did worse than random subsampling, and few-shot prompting performed at or below the no-augmentation baseline. Our hypothesis is that plain SBERT similarity lands on same-topic neighbors. Masked, cross-event retrieval should avoid that failure. If it does not, random donors will win our ablation and we will report that outcome.
Research Program
Pools. For each training event, we build a pool of the facts the event's posts actually assert, extracted by the LLM from that event's true and non-rumor posts. Separately, we build a donor index. Every training post is stored in an entity-masked form: named entities, numbers and hashtags are replaced by typed placeholders. Each masked post is embedded with a retrieval encoder.
Generating a transplant. To create a post with target label $y'$ for event $A$, we query the donor index with masked posts from $A$. Retrieval is restricted to donors whose event differs from $A$ and whose label is $y'$. We then give the generator LLM three inputs: the donor post, the facts of event $A$, and a label-specific instruction. The instructions are:
- Non-rumor: comment on the event.
- True rumor: assert a claim consistent with $A$'s known facts.
- False rumor: assert a specific claim that contradicts them.
- Unverified: assert a claim that the fact pool neither supports nor refutes.
The output keeps the donor's rhetorical form and claim type and takes its content from $A$.
Checking labels. A natural language inference model checks each transplant's claim against $A$'s fact pool, and transplants whose inference verdict disagrees with the intended label are discarded. A human-annotated sample of transplants estimates the remaining label noise.
Avoiding a generator shortcut. We generate transplants for every label within every event, including the event's majority label. Otherwise "sounds LLM-written" would itself become a label cue.
Training. We train an ordinary text classifier on the real posts plus the transplants. We start with DeBERTa, the classifier used by RumorLLM. We also test transplants as the demonstration pool for an in-context LLM detector.
Retrieval encoders. We compare three: SBERT on raw text, SBERT on entity-masked text, and a stance/style encoder trained to predict label while being unable to predict event. The third is a GPT-RE-style task-specific representation.
Baselines
Naive reference. A DeBERTa classifier trained on the unaugmented event-separated training set. We add a variant trained with rule-based cross-event entity swapping, which replaces event $A$'s names and places with those of other events at the same rate as our transplants. This variant isolates what the LLM contributes beyond lexical substitution.
Existing augmentation. RumorLLM-style keyword-anchored generation and plain LLM paraphrasing, given the same number of generated posts and the same generator model. These test whether any extra data helps, or only event-balanced data. RumorLLM reports 82.55% accuracy on BuzzFeed and 93.91% on PolitiFact, but only under random splits. We found no reported result on Lacuna for RumorLLM under event-separated splits, so we will run it ourselves.
Representation-level debiasing on the same text encoder. A gradient-reversal event discriminator in the style of MetaDetector, plus FNDCD-style sample reweighting. Both are tuned with the same hyperparameter budget as our method, measured in number of trials and GPU hours. MetaDetector reports 72.19% average accuracy on PHEME events and 74.80% on its Weibo cross-event split, under its own protocol.
Strongest existing method. FADE, which reports event-separated accuracy of 71.81% on Twitter15, 77.72% on Twitter16 and about 60% on PHEME. FADE uses propagation graphs, while our base classifier sees only text, so we compare in two ways:
- text-only transplant models against FADE's reported numbers;
- FADE rerun with its BERT node encoder trained on transplant-augmented text.
Beating FADE means one of two things. Either the text-only model matches or exceeds its accuracy, or transplants added to FADE improve on its reported figures by more than the seed-to-seed spread.
Publisher Style Aggregation (MT+PSA) is reported as 19.00 and 20.61 points above the best baselines on Twitter15 and Twitter16. It is included for completeness but flagged as using publisher history, which our setting does not assume.
Ablations. These isolate the key ingredient: random cross-event donors, raw-SBERT donors without the cross-event constraint, and transplants generated for minority labels only.
Why not just…?
Why not just mask or swap entities with rules? Jiang et al. report that masking event keywords degrades standard detectors, because it removes the context that carries stance and affect. Swapping entities across events is cheaper and harder to dismiss. It keeps the post fluent, and it also weakens the name–label link. But swapping cannot create a false-rumor post for an event that only has true posts: the claim type and assertion stay those of the original post. It therefore leaves the event–label skew over claim types in place. We treat this as a real threat. The entity-swap baseline is matched in volume. If it recovers most of the transplant gain, the LLM step is not worth its cost and we will say so.
Why not just debias the representation adversarially or by reweighting? MetaDetector itself documents negative transfer under plain adversarial training, and needed extra importance weighting to limit it. FNDCD down-weighting and FADE's bias subtraction cannot supply labels that an event lacks. The comparison on a shared text encoder settles this. We also report probe accuracy for event identity on the frozen features of each model. That shows whether transplants remove event information as effectively as adversarial erasure, and whether they do it while raising task accuracy.
Why not just prompt a strong LLM detector with SBERT-retrieved demonstrations? LM-BFF and Margatina et al. show that nearest-neighbor demonstrations beat random ones. On these benchmarks, however, nearest neighbors come from the same topic, which recreates the shortcut through the prompt. There is also a confound. PHEME's events date from the mid-2010s and are probably known to modern LLMs, so a zero-shot LLM may succeed by recalling outcomes rather than by reasoning. We will run this baseline with raw-SBERT and with masked-transplant demonstration pools. To separate genuine generalization from memorized event outcomes, we will also check it on a recent event collection gathered after the LLM's training cutoff.
Proposed Experiments
Benchmarks and protocol. We use Twitter15 and Twitter16 under Wu and Hooi's event-separated splits, reporting accuracy and macro-F1 over the four classes. On PHEME we use leave-one-event-out evaluation. As a secondary transfer test, we train on Twitter data and test on the COVID-19 collection used by FNDCD, which reports a 16.37% fake-news F1 gain there. Every configuration runs over five seeds.
Decisive checks. Four results decide the outcome:
- The masked cross-event donor must beat random donors and raw-SBERT donors. This directly tests whether retrieval earns its place, given the Bangla result.
- The transplant model must beat LLM paraphrasing at equal volume. Without that, "more data" explains the gain.
- Event-probe accuracy on the classifier's features must fall relative to the unaugmented model. We also report the empirical $I(E;Y)$ of each augmented training set, and expect accuracy gains to track reductions in it across ablations.
- On the human-annotated transplant sample, label agreement must be high enough that noise cannot explain the results.
We also run a stress test that trains only on events whose majority label differs from the test event's. This is where representation-level methods have the least to work with and transplants should help most.
If transplants help only when paired with FADE's graph signal, or if rule-based swapping matches them, the paper's conclusion becomes a narrower claim about data balance rather than generation. That is still a useful answer.
References
- Probing Spurious Correlations in Popular Event-Based Rumor Detection Benchmarks
- RumorLLM: A Rumor Large Language Model-Based Fake-News-Detection Data-Augmentation Approach
- Cross-Topic Rumor Detection using Topic-Mixtures
- Each Fake News Is Fake in Its Own Way: An Attribution Multi-Granularity Benchmark for Multimodal Fake News Detection
- MetaDetector: Meta Event Knowledge Transfer for Fake News Detection
- Evolving to the Future: Unseen Event Adaptive Fake News Detection on Social Media
- Unseen Fake News Detection Through Casual Debiasing
- Iterative Counterfactual Data Augmentation
- GPT-RE: In-context Learning for Relation Extraction using Large Language Models
- Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation
- Capturing the Style of Fake News
- Learning Disentangled Latent Topics for Twitter Rumour Veracity Classification
- Addressing Data Scarcity in Bangla Fake News Detection: An LLM-Based Dataset Augmentation Approach
- Making Pre-trained Language Models Better Few-shot Learners
- Active Learning Principles for In-Context Learning with Large Language Models