Labeling news articles for political bias, hyperpartisanship, or reliability one at a time is expensive. The common shortcut is to give every article the label of the outlet that published it. This is distant supervision: cheap labels inherited from a coarse source. The resulting labels are wrong in a structured way. Propagandistic outlets mix in factual wire reports. Low-credibility sites republish ordinary journalism. Biased outlets sometimes run pieces that go against their usual slant. Neural classifiers trained on such labels learn to recognize the outlet rather than the bias. A RoBERTa model fits randomly assigned site labels with 95.04% accuracy, and its accuracy drops from 96.94% to 80.36% on unseen sites (Hidden Biases in Unreliable News Detection Datasets). The current fixes throw most of the data away. One SemEval team kept 32,000 of 750,000 articles (Team yeon-zi).
We propose to correct publisher labels with graph-based label propagation, with one structural change. Label propagation is a classical semi-supervised method: build a graph connecting similar documents, then let labels spread along the edges so that close neighbors end up with similar labels. In our version, edges are allowed only between articles from different publishers. An article's corrected label is then decided by how other outlets' articles on the same content are labeled, never by its own outlet's other articles. A wire story republished in both a hyperpartisan and a mainstream outlet ends up with mixed, uncertain neighbors, which flags it. A genuinely hyperpartisan op-ed sits near other hyperpartisan outlets' op-eds and keeps its label. In short, the method corrects publisher labels by asking what other publishers' similar articles say, and it does so through graph diffusion so that the full corpus and a small gold seed set can be used without discarding anything.
Core Claim
When label noise is tied to the source, the relevant failure in graph propagation is not noisy edges or noisy seeds in general. It is that edges within a publisher carry the publisher's label to all of its articles. Ordinary k-nearest-neighbor graphs built on text embeddings will be dense inside each publisher, because embeddings encode outlet style. That style is exactly what the random-site-label result exposes. Propagation on such a graph simply reproduces the publisher labels.
Removing within-publisher edges changes what propagation can do. It can no longer carry a label that is supported only by an outlet's identity. It can still carry a label that is supported by content shared across outlets. We claim that this restriction, combined with per-publisher reliability weights and a small clamped gold set, recovers article-level labels better than three existing approaches: hard filtering, source-adversarial representation learning, and standard weak-supervision label propagation. We also claim it does so on source-disjoint evaluation, which is the setting that matters.
Relation to Prior Work
The closest existing work is Label Propagation with Weak Supervision. It attaches weak labelers to the data graph as virtual "dongle nodes" whose edge weights reflect each labeler's estimated accuracy (LPAD). On WRENCH it lifts YouTube accuracy from about 82% to 90.32% and SMS coverage from about 9% to nearly 100%. We reuse its idea of representing a weak source as a weighted anchor rather than a hard label. Our case differs in three ways. First, its weak sources are keyword and regex rules. Second, its error analysis assumes error is roughly uniform within a neighborhood. Third, it builds a fixed k-nearest-neighbor graph. With publishers as the weak sources, that fixed graph is exactly what spreads outlet identity, and the uniform-error assumption fails, because errors concentrate inside publisher clusters.
Label Propagation for Graph Label Noise (R²LP) rebuilds a graph to restore homophily, meaning neighbors tend to share labels. It then rectifies labels by propagation and stays stable under 60% random label noise on citation and social graphs. It supplies the iterate-reconstruct-rectify template, but it was tested only on uniform synthetic flips. Under source-level noise, rebuilding the graph from current label estimates would strengthen within-publisher cliques, since all of a publisher's articles share an estimated label.
On the Equivalence of Decoupled Graph Convolution Network and Label Propagation shows that decoupled graph networks are equivalent to propagating labels and then training a classifier on the soft labels. We adopt that "propagate, then train" design so the final model is an ordinary text classifier that needs no graph at test time. Graph Attention Multi-Layer Perceptron shows that propagated labels can leak an example's own label back to it. We exclude each gold seed from its own propagated target for the same reason.
On the noise side, several papers motivate the problem. A Survey on Computational Propaganda Detection and Capturing the Style of Fake News document that outlets mix content types. We Can Detect Your Bias measures about 3% of AllSides articles defying their outlet's label. That paper is also the main representation-learning competitor. Its triplet pre-training groups same-ideology articles from different outlets, lifting Macro F1 on unseen media from 35.53 to 48.26, and it needs Twitter follower data to reach 64.29. Our cross-publisher edges use a similar idea, that agreement across outlets should count, but we apply it to the labels through diffusion rather than to the embedding. The two are complementary, and we test them together. Bias Bubbles and Team yeon-zi both apply hard agreement filters article by article, which is what we replace. Optimal Graph Learning and Nuclear Norm Maximization learns separate within-domain and cross-domain graphs for domain adaptation. Our design is similar in spirit, treating each publisher as a domain, but it assumes clean source labels and its cost grows quadratically with corpus size.
Research Program
Graph construction. We embed every article with a frozen news-domain encoder. Before building the graph, we remove the directions in embedding space that best predict publisher identity, using a linear probe and projecting out its weight directions. This cheap residualization targets the outlet fingerprint. We then connect each article to its $k$ nearest neighbors among other publishers only. We also exclude publishers from the same owner network, when that is known, so that sibling outlets cannot reintroduce the shortcut.
Propagation. We solve one sparse, convex smoothing problem for soft label vectors $F_i$:
$$ \min_F \sum_{p(i)\neq p(j)} W_{ij}\,\|F_i - F_j\|^2 \;+\; \lambda \sum_i a_{p(i)}\,\|F_i - y_{p(i)}\|^2 \;+\; \mu \sum_{i \in G} \|F_i - y_i\|^2 . $$
Here $p(i)$ is the publisher of article $i$, and $W_{ij}$ is the cross-publisher similarity weight. $y_{p(i)}$ is the publisher's distant label, entering as a dongle-node anchor, and $a_{p}\in[0,1]$ is that publisher's reliability. $G$ is the small human-labeled seed set with gold labels $y_i$, and $\lambda,\mu$ trade off the three terms. The first term only couples articles from different outlets, so a publisher's label reaches its own articles only through the anchor term, at strength $\lambda a_p$. We estimate $a_p$ from how often the publisher's articles agree with their cross-publisher propagated labels in a first pass, plus agreement on gold seeds when a publisher has any. We alternate estimating $a_p$ and solving for $F$ for a few rounds, following R²LP's iterative structure but without rebuilding the graph from label estimates. Each solve is a sparse linear system, so it scales to the 750,000-article SemEval corpus.
Training. We fine-tune a RoBERTa classifier on all articles, using the soft targets $F_i$ weighted by their confidence. No article is discarded.
Analysis. Gap 4 of the research file notes that existing flow-based error bounds fail for unseen publishers, where no labeled information flows into the test cluster. Our construction makes the relevant quantity explicit: the cross-publisher inflow into each publisher's articles. We will report accuracy of the corrected labels as a function of this inflow. The goal is to identify when the method is expected to fail, such as niche outlets covering topics nobody else covers, rather than to prove a new bound.
Baselines
The do-nothing reference is RoBERTa fine-tuned directly on all publisher labels. On source-disjoint NELA it reaches 80.36% accuracy (Hidden Biases), and on SemEval Team yeon-zi report that training on the full noisy set was worse than filtering.
The strongest existing filters are the following. Team yeon-zi's confidence filter reports 75.8% accuracy on the SemEval-2019 Task 4 by-article test set (Team yeon-zi). Bias Bubbles' overlap selection reports F1 0.803 against 0.780 for gold-only training (Bias Bubbles). Team QCRI-MIT's tuned n-gram logistic regression reports 76.0% accuracy (Team QCRI-MIT).
The strongest representation-learning competitor is triplet pre-training, which reports Macro F1 48.26 on the AllSides media split without external data (We Can Detect Your Bias). The 64.29 figure that uses Twitter is reported as an upper reference, not as a fair comparison.
The closest graph competitors are LPAD on an unrestricted k-nearest-neighbor graph, with publishers as dongle nodes (Label Propagation with Weak Supervision), and R²LP run on the same article graph. Neither has a reported result on news corpora on Lacuna, so we run both ourselves.
Every method gets the same encoder, the same gold seeds, the same final classifier, and an equal hyperparameter search budget tuned on a source-disjoint validation split. To count as beating these baselines, we must exceed 75.8% accuracy and 0.803 F1 on SemEval by-article and 48.26 Macro F1 on the AllSides media split. We must also beat unrestricted LPAD by a margin larger than the seed-to-seed standard deviation.
The ablations isolate the key ingredient. The most important one restores within-publisher edges and changes nothing else. Others drop the reliability weights $a_p$ (setting them to a constant), drop the publisher-direction projection, and replace diffusion with a single-hop vote.
Why not just…?
Why not just use stylometric features that describe how an article is written? Team QCRI-MIT found that readability and vocabulary-richness features helped on clean by-article data but hurt when the training labels were publisher-level (Team QCRI-MIT). Brenda Starr's surface-feature classifier scored 0.596 by-article F-score against 0.712 for their neural model (Brenda Starr). Style features are themselves an outlet fingerprint, so they do not address the label problem.
Why not just filter disagreeing articles, as Team yeon-zi and Bias Bubbles do? Filtering keeps only articles where a weak model already agrees with the publisher label. It therefore systematically removes the cases that matter most: neutral pieces in biased outlets. It also caps the data at under 10% of the corpus. Our method keeps every article and relabels the disagreeing ones rather than dropping them. The comparison above against 75.8% and 0.803 tests directly whether keeping and relabeling helps.
Why not just make embeddings publisher-invariant with adversarial training or triplet loss? This is a real competitor, and we run it. Its reported ceiling without external metadata is 48.26 Macro F1 on unseen outlets. Making the embedding invariant also leaves the labels unchanged, so a classifier trained on the invariant embedding is still fit to the wrong labels for the wire-copy articles. Our design is compatible with it. One arm runs our propagation on triplet-pretrained embeddings, and if that combination wins, the gains add rather than overlap.
Why not just do grouped cross-fitting: train on all other publishers, predict the held-out publisher, and flag disagreements? This is the most serious threat, because it implements the same leave-the-publisher-out principle without a graph. We include it as a baseline with matched compute: one classifier per publisher fold, with folds grouping sibling outlets. The graph version has two possible advantages. It uses the gold seeds transductively, so their information spreads through the whole graph rather than only into the classifier's weights. It also costs one sparse solve instead of dozens of fine-tuning runs. If cross-fitting matches our method on accuracy, we will report that the principle, not the graph, is the contribution, and recommend the cheaper of the two. The single-hop vote ablation makes the same test against the multi-hop part of propagation.
Proposed Experiments
The main benchmarks are SemEval-2019 Task 4 and AllSides. On SemEval we use the by-publisher training corpus, the 645 crowd-labeled articles as seeds, and the by-article test set, measuring accuracy and F1. On AllSides we use the media-based split, which has 34,737 articles from 73 outlets, and measure Macro F1 and mean absolute error. On AllSides we hide article labels, replace them with each outlet's majority label, and keep a small gold subset as seeds. Because the true article labels are known, we can measure precision and recall at recovering the roughly 3% of articles that defy their outlet.
We add NELA with a source-disjoint split to compare against the 80.36% reference.
The decisive diagnostic reuses the random-site-label probe from Hidden Biases. We give sites random labels and run each label-correction method. Standard and LPAD-style propagation should mostly reproduce the random labels, since the within-publisher edges carry them. Our restricted graph should drive the corrected labels toward chance, because content shared across outlets does not support them. A classifier trained on our corrected labels should also lose most of the 95.04% fit. If our method instead preserves the random labels, the outlet shortcut is getting through the cross-publisher edges, through sibling outlets or residual style. That outcome would falsify the core claim.
Finally, we stratify errors by each publisher's cross-publisher inflow, and by topic using the topic tags in AllSides. This shows where the method helps and where low-inflow outlets remain uncorrectable.
References
- Hidden Biases in Unreliable News Detection Datasets
- Team yeon-zi at SemEval-2019 Task 4: Hyperpartisan News Detection by De-noising Weakly-labeled Data
- Label Propagation with Weak Supervision
- Resurrecting Label Propagation for Graphs with Heterophily and Label Noise
- On the Equivalence of Decoupled Graph Convolution Network and Label Propagation
- Graph Attention Multi-Layer Perceptron
- A Survey on Computational Propaganda Detection
- Capturing the Style of Fake News
- We Can Detect Your Bias: Predicting the Political Ideology of News Articles
- Bias Bubbles: Using Semi-Supervised Learning to Measure How Many Biased News Articles Are Around Us
- Optimal Graph Learning and Nuclear Norm Maximization for Deep Cross-Domain Robust Label Propagation
- Team QCRI-MIT at SemEval-2019 Task 4: Propaganda Analysis Meets Hyperpartisan News Detection
- Brenda Starr at SemEval-2019 Task 4: Hyperpartisan News Detection