Text-Matched Training for Rumor Detectors That Read the Crowd

Rumor detectors that look at the whole social conversation around a claim still mostly read the claim itself. When they are tested fairly, by training on older rumors and testing on newer ones, they fall apart. A graph model over reply trees that scores 83.8% on a random split of Twitter15 drops to 41.5% on a forward chronological split, which is below chance for four classes. The authors of that study trace the failure to memorized story vocabulary and to over-reliance on the source post, even in models built to use context (Examining the Limitations of Computational Rumor Detection Models Trained on Static Datasets). The replies, the skepticism, and the "why hasn't this been deleted?" comments are exactly the signals that should transfer to new stories. They go unused because, during training, the source text already explains most of the label, so the gradient never rewards the crowd.

We propose to take that shortcut away during training only. A separate text-only model, fit out-of-fold, estimates how predictable each cascade's label is from its source post. This plays the role of a propensity score: the probability of the observed outcome given the confounding input. The conversation model is then trained with weights that balance labels across levels of text-predictability, so it learns only from what the crowd adds beyond the text. At test time the text score and the crowd increment are added. The same construction runs over time: the prediction made from the prefix up to an earlier moment acts as the "admission score", and each new window of replies is trained to supply an update to it. This is how clinical risk models moved from a single intake score to trajectories (Optimal intensive care outcome prediction over time using machine learning).

Core Claim

Balancing on the text score during training will make conversation-based rumor detectors hold up far better under chronological and event-held-out evaluation than models trained without it. It will also make rolling predictions flip less often, because the crowd component moves only when new replies carry information the earlier prefix did not. Concretely, we write $\hat e_k(x_i)$ for the out-of-fold probability that cascade $i$ has class $k$ given only its source text $x_i$. The context branch $f$, which sees the reply tree $g_i$, is trained on

$$\mathcal{L} = \sum_i w_i \, \ell\big(\operatorname{softmax}(\log \hat e(x_i) + f(g_i)),\, y_i\big), \qquad w_i \propto \min\big(1/\hat e_{y_i}(x_i),\, c\big),$$

where $\ell$ is cross-entropy, $y_i$ is the true class, and $c$ is a truncation cap that limits the variance of extreme weights. Holding $\log \hat e$ fixed as an offset means $f$ is paid only for correcting the text. The weights push up cascades whose text points the wrong way, and these are exactly the cases where the crowd must carry the decision. We use truncated weighting rather than hard 1:1 matching because Twitter15 and Twitter16 hold only on the order of a thousand cascades each, and matching would throw away too much data.

Research Program

The closest prior work is Improving Generalizability of Fake News Detection Methods using Propensity Score Matching. That paper treats each word as a treatment, matches documents on the other words, and keeps words with a causal effect on veracity, which raises cross-domain AUROC from 0.56 to 0.68 (PolitiFact to GossipCop). It motivates us but runs in the opposite direction. It balances text in order to select text features, whereas we balance on text in order to train a model of the conversation, and it never touches social context or time. Causal Understanding of Fake News Dissemination on Social Media supplies the reweighting technique: it applies inverse propensity weighting to correct exposure bias when learning user sharing embeddings. We borrow the estimator, not the target, since that work explains user susceptibility and does not predict veracity. Propensity Score Matching for Causal Inference with Relational Data constrains the design. It shows that underspecifying confounders causes far worse spurious effects than including too many. We therefore make the text propensity model rich, a fine-tuned transformer over the source post and publisher identity, rather than a sparse bag of words. A weak propensity model would leave story leakage in place and make the method look better on paper than it is.

The context branch reuses the bi-directional reply-tree encoder of Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks, with one deliberate change. We remove its root-feature enhancement, which copies the source post's representation into every node. Song et al. identify that design as a route to the memorization trap, and our offset already supplies the text. We also build a sequential variant on the hierarchical transformer that Song et al. found robust on Weibo.

For the rolling version, time is cut into windows. At window $t$ the offset becomes the frozen, out-of-fold prediction from the prefix up to $t-1$, and the weights balance labels on that prefix prediction. This is the rumor analogue of a time-varying hazard update (Real-time Prediction of COVID-19 related Mortality using Electronic Health Records), where risk is re-estimated as each new measurement arrives. A second weight handles a problem clinical work already names, survivor attrition (Optimal intensive care outcome prediction over time using machine learning). Cascades that stop receiving replies drop out of late windows non-randomly, so we multiply in the inverse probability of a cascade still being active at $t$, estimated from its prefix. This complements the stopping rule of Early Rumor Detection Using Neural Hawkes Process with a New Benchmark Dataset. HEARD models when predictions flip in order to decide when to stop, while we aim to make the underlying rolling predictor flip less in the first place, and the two can be stacked.

Multi-task learning enters as the main competing way to achieve balance, not as the core mechanism. The alternative is a shared encoder with a veracity head plus an adversarial head that tries to recover the text score or the story cluster from the conversation representation. The two objectives pull against each other, which is the conflict RotoGrad: Gradient Homogenization in Multi-Task Learning addresses with per-task rotations and gradient-magnitude equalization. RotoGrad was shown on vision benchmarks such as NYUv2, never on graph encoders or on invariance-versus-discrimination conflicts. We run it here as a serious competitor.

Baselines

The naive reference points are a text-only transformer on the source post and an unweighted Bi-GCN trained normally. For the Bi-GCN, Song et al. report 41.5% accuracy on Twitter15 under a forward chronological split (83.8% random). The strongest existing conversation model on chronological splits is the hierarchical transformer of Song et al., reported at around 94% accuracy on Weibo under chronological splits. We found no reported result on Lacuna for H-Trans on Twitter15 or Twitter16 chronological splits, so we will produce it ourselves. For rolling prediction, the strongest method is HEARD, which reports a stabilized early accuracy of 0.789 on TWITTER against 0.620 for the next best baseline, and reaches stable decisions in under six hours on PHEME.

Every model will use the same forward chronological splits and the same reply prefixes, pretrained text encoders, and hyperparameter search budget. The splits follow Song et al. on Twitter15, Twitter16, Weibo and Sun-MM, plus leave-one-event-out on PHEME and HEARD's protocol on BEARD. Selection uses a validation slice that is itself later in time than training. Beating the baselines means a macro-F1 gain on the Twitter chronological splits that exceeds seed-to-seed variation across five seeds, without losing more than a point on the random split. On BEARD and PHEME, it means a higher stabilized early accuracy than our rerun of HEARD at equal or lower flip counts.

Two ablations isolate the key ingredient. The first keeps the offset but sets all weights to one, which is plain residual fitting on top of the text score. The second replaces the out-of-fold text propensity with the model's own in-sample confidence, which amounts to focal-style hard-example upweighting. If either ablation matches the full method, then the separately estimated, balanced propensity is not what matters.

Why not just…?

Why not just delete the source text and train on the conversation alone? This is the most serious threat, since Song et al. show the source text is where memorization lives. A conversation-only model does discard real signal, though, most of all in the earliest windows, when there are few replies. The comparison will show this directly. We expect conversation-only to match us on late, reply-rich prefixes and lose clearly in the first hours. If it matches us everywhere, we will report that the text offset is unnecessary.

Why not just fit the conversation model on the text model's residuals, or combine the two as a product of experts? Our unit-weight ablation is exactly this, so the question is answered empirically. The reason to expect a gap is that residual fitting still spends most of its gradient on cascades the text already gets right. Balancing reallocates that gradient toward cascades where the text misleads, which are the ones that resemble new stories.

Why not just train a multi-task model with an adversarial invariance head, balanced with RotoGrad? This is a legitimate alternative that does the balancing inside the representation rather than in the data weights, and it is run at equal budget. Adversarial invariance to a label-correlated quantity is known to be unstable, and weighting is simpler to audit. But no evidence in the file settles which works better on graphs, so a tie would be a publishable finding in itself.

Why not just prompt or fine-tune a large language model on the claim? Song et al. find that accuracy of text models tracks train–test text overlap, measured by IoU and DICE similarity. A text-only fine-tuned model is therefore among our baselines, and we expect its chronological accuracy to fall with the other text-reliant systems.

Decisive Experiments

The central test reruns Song et al.'s memorization diagnostic. For each test cascade we compute its maximum text overlap with training stories and plot accuracy against it. The prediction is that the unweighted Bi-GCN's accuracy rises steeply with overlap, while the balanced model's curve is flatter, especially in the low-overlap bin where new stories live. The result that would kill the idea is a flat gain that disappears in that low-overlap bin.

Second, on BEARD and PHEME we measure how often predictions flip per cascade and stabilized early accuracy as a function of time. We compare the windowed offset model with and without the attrition weight, and with HEARD's stopping rule placed on top of each.

Third, a controlled check on the propensity model follows Arbour et al.'s warning. We degrade the text propensity model from the full transformer to a small bag of words. If chronological gains shrink as the propensity weakens, that supports the causal reading, because the method works to the extent it actually absorbs the text. If gains persist with a useless propensity model, the improvement is ordinary regularization, and we will say so.