Most fake news detectors are scored on benchmarks where fake and real stories appear in roughly similar numbers. On a real platform, misinformation is a small fraction of what gets posted. At that rarity a detector can look excellent while catching almost nothing. We propose a cheap way to make text classifiers aware of social structure, together with an evaluation protocol that shows whether the gain survives realistic rarity. The mechanism modifies TF-IDF (term frequency–inverse document frequency), the classic weighting that boosts words that are frequent in a document but rare across the whole collection. Standard TF-IDF treats the collection as one well-mixed pool, and that is exactly the assumption an idealized, fully mixed network makes. On real platforms, text is written inside tight communities. A phrase can be everywhere inside a fringe group and almost absent everywhere else. Global TF-IDF blurs that pattern. We instead weight each term by how concentrated it is in the author's community compared with the rest of the network. Community memberships come from the interaction graph, not from veracity labels.
The two-sentence version is as follows. Word weights computed from which communities use a word give a lexical classifier structural information without needing labels, which matters when true positives are scarce. We test whether this beats both plain TF-IDF and graph neural networks once fake news is made rare and performance is measured with metrics that a trivial "everything is real" model cannot game.
Core Claim
Let $df_c(t)$ be the fraction of documents from community $c$ that contain term $t$, and let $df_{\neg c}(t)$ be the same fraction over all other communities. We define the community-localization score of $t$ for $c$ as a smoothed log-ratio, with smoothing constant $\alpha$:
$$\rho(t,c) = \log\frac{df_c(t)+\alpha}{df_{\neg c}(t)+\alpha}.$$
A document $d$ whose author (or early engagers) belongs to communities with mixture weights $\pi_{d,c}$ receives two feature blocks. The first is ordinary TF-IDF. The second is term frequency multiplied by $\sum_c \pi_{d,c}\,\rho(t,c)$.
This score has the same form as the class log-count ratio in NB-SVM (a naive-Bayes-weighted linear SVM). NB-SVM computes that ratio from labels. We compute it from communities, which cost no labels. That difference is the point. When positives are rare, label-derived term weights are estimated from very few examples. Community-derived weights are estimated from the whole unlabeled graph.
The claim is falsifiable in two ways. First, on synthetic networks the gain from $\rho$ should shrink as networks become better mixed, and should vanish on Erdős–Rényi or fully connected graphs, where local exposure equals global frequency. Second, on real benchmarks the gain should grow as fake-news prevalence falls.
Research Program
The work has three parts.
Controlled synthetic study. We borrow the scale-free community networks of the public goods study, which vary the inter-community link probability $q$ and show that dense intra-community ties change the dynamics qualitatively. We plant community-specific vocabularies and seed misinformation cascades from one fringe community. We then compare against Erdős–Rényi and fully connected controls, the kind of idealized substrates used by Zhao et al. and Spike et al.. Here idealized structures serve as a measurement tool: they tell us when community-localized weighting should help and when it should not. This responds to the warning, attributed to Schoenebeck, that simplified topologies can create artifacts.
Real benchmarks with communities from the interaction graph. For PolitiFact and GossipCop from FakeNewsNet, we build a user co-engagement graph from retweets and replies. We run standard modularity-based community detection. We compute $\rho$ from users' historical posts, the same text source that UPFD embeds with BERT. For spreader detection on FACTOID, communities come from the Reddit reply-thread graph. Community detection is always run on the training-period graph only, to avoid leakage. We also run an early-detection variant in which the mixture $\pi_d$ is estimated from only the first $k$ engagers. This targets the retrospective-cascade limitation noted in the graph fake news survey.
Rarity-controlled evaluation. Test sets are subsampled so that fakes (or spreaders) make up 50%, 10%, 5% and 1% of items, with repeated random draws. Training is done at matched prevalence. We report the Matthews correlation coefficient (MCC), which uses all four confusion-matrix cells. We also report $F_1$-mult, the product of the per-class $F_1$ scores proposed by Logacheva et al.. They showed that minority-class $F_1$ alone rewards an all-positive predictor, and that $F_1$-mult resists this. Finally we report precision at a fixed recall of 50%. Accuracy and AUROC are reported only to demonstrate their failure. Rajkomar et al. and Ruiz Puentes et al. show that AUROC hides large numbers of false positives when negatives dominate. Every metric table includes all-real and all-fake synthetic predictors, following the gaming test of Logacheva et al.
Baselines
Naive reference. The majority-class predictor, plus an all-positive predictor. At 1% prevalence the majority predictor gets 99% accuracy and zero MCC. If any reported configuration fails to clearly beat it on MCC, that configuration is uninformative.
Plain TF-IDF. A logistic regression or linear SVM over word and character n-gram TF-IDF. This is the setup that Nayel and Andrew found competitive for lexical detection tasks. Our method must beat this to show that the community term adds anything.
NB-SVM. Label-derived log-ratio weighting, as used by WUY in its BERT+NB-SVM hybrid. This is the most direct competitor to our label-free ratio.
Strongest existing method. UPSR, which reconstructs missing propagation edges using Gaussian node distributions and runs dual graph convolutional networks. It reports 0.914 accuracy and 0.910 F1 on PolitiFact, and 0.977 accuracy and 0.976 F1 on GossipCop. The file gives absolute numbers only for UPSR. For UPFD it reports only that UPFD beats GCNFN by about 1% accuracy and F1. On FACTOID, the strongest reported result is a graph attention network (GAT) over User2Vec embeddings at 61.6% F1. RumorLLM reports 93.91% accuracy, 0.8679 F1 and 0.9233 AUC-ROC on PolitiFact. We found no reported result on Lacuna for any of these detectors at prevalence of 5% or below, or scored with MCC or $F_1$-mult. The low-prevalence comparison is therefore new measurement.
Fairness of comparison. All systems see identical splits, the same users' historical text, and the same graph. Each gets the same hyperparameter search budget, tuned on a validation set at the target prevalence. Decision thresholds are tuned for MCC on validation data for every model.
Key ablations. These isolate the ingredient we propose. First, TF-IDF plus a one-hot community indicator, which tests whether community identity alone explains any gain. Second, $\rho$ computed on randomly shuffled communities that keep the original community sizes. Third, $\rho$ computed from communities detected on a degree-preserving rewired graph.
What counts as winning. On PolitiFact and FACTOID at 1–5% prevalence, the method should beat plain TF-IDF and the one-hot ablation on MCC, with bootstrap confidence intervals that do not overlap. It should also match or beat UPSR and UPFD on MCC at a small fraction of their compute. At 50% prevalence, we only require that it not trail UPSR by more than a few points.
Why not just…?
Why not just add community membership as a feature? This is the most serious threat. A linear model with one-hot community indicators can only shift its decision threshold per community. It cannot change how much each word counts inside a community. A model with explicit community-by-term interaction features could learn that reweighting, but only from labels, and at 1% prevalence those labels are scarce. The one-hot ablation and an interaction-feature ablation test this directly. If either matches our method at low prevalence, the reweighting idea is not needed, and we will report that.
Why not just use the graph neural network? The survey and FACTOID report that social-context models beat text-only models by 5–10 points of accuracy. Those numbers come from near-balanced splits and are scored by accuracy. It is untested whether a GNN's relational features, learned from labels, remain useful when positives are rare, or whether they overfit. The rarity sweep settles this. If UPSR keeps its lead on MCC at 1% prevalence, the practical case for our method narrows to cost and early detection, and we will say so.
Why not just rebalance with augmentation or an MCC loss? RumorLLM's synthetic rumors improve balanced-split scores. However, WUY found that simple augmentation still overfit. Synthetic text also carries no community context. The differentiable MCC loss of Márquez Herrera et al. is complementary to our method, not a replacement: it changes the training objective, not the features. We therefore apply loss reweighting, and the MCC loss where the model is differentiable, to every baseline. This keeps the comparison about the features.
Proposed Experiments
Synthetic check. On scale-free community networks, the MCC gain from $\rho$ over plain TF-IDF should decrease as $q$ increases. It should not be distinguishable from zero on Erdős–Rényi and fully connected controls. If the gain appears on well-mixed graphs, the method is exploiting something other than community localization and the core claim fails.
Real-data prevalence sweep. Plot MCC, $F_1$-mult and precision at 50% recall against prevalence for every baseline on PolitiFact, GossipCop and FACTOID. The expected signature has two parts. Accuracy and AUROC should stay high for all methods, including the trivial ones. MCC should separate the methods, with our method's advantage over plain TF-IDF widening as prevalence falls.
Early-detection test. Estimate $\pi_d$ from only the first 5, 10 and 20 engagers. This asks whether community-localized weighting keeps its value before full cascades exist, the regime where the propagation-based detectors described in the survey have little to work with.
Mechanism checks. The shuffled-community control and the rewired-graph control must both erase the gain. If they do not, the gain is not coming from real community structure.
References
- The evolution of cooperation in public goods games on the scale-free community network under multiple strategy-updating rules
- Analyzing Resilience of Interest-Based Social Networks Against Node and Link Failures
- Minimal Requirements for the Emergence of Learned Signaling
- FakeNewsNet: A Data Repository with News Content, Social Context, and Spatiotemporal Information for Studying Fake News on Social Media
- User Preference-aware Fake News Detection
- FACTOID: A New Dataset for Identifying Misinformation Spreaders and Political Bias
- Fake News Detection Through Graph-based Neural Networks: A Survey
- Metrics for Evaluation of Word-level Machine Translation Quality Estimation
- Reply: metrics to assess machine learning models
- PharmaNet: Pharmaceutical discovery with deep recurrent neural networks
- NAYEL at SemEval-2020 Task 12: TF/IDF-Based Approach for Automatic Offensive Language Detection in Arabic Tweets
- JudithJeyafreedaAndrew@DravidianLangTech-EACL2021:Offensive language detection for Dravidian Code-mixed YouTube comments
- WUY at SemEval-2020 Task 7: Combining BERT and Naive Bayes-SVM for Humor Assessment in Edited News Headlines
- Uncertainty-aware Propagation Structure Reconstruction for Fake News Detection
- RumorLLM: A Rumor Large Language Model-Based Fake-News-Detection Data-Augmentation Approach
- Semantic Segmentation of 3D Medical Images with 3D Convolutional Neural Networks