Where Disagreement Lives in a News Article

When people label sentences in news articles as biased or neutral, they disagree a great deal. Crowd workers labeling biased sentences in news reached negative agreement (Annotating and Analyzing Biased Sentences in News Articles using Crowdsourcing). Most of this research shows annotators a sentence on its own. Most models are then trained to predict one "correct" label per sentence. Real readers never meet a sentence alone, though. They meet it under a headline, after a lede (the opening paragraph that frames the story), and often inside a quote attributed to some speaker. This proposal asks a simple question: how much of the disagreement comes from what each annotator could see, and how much comes from who the annotator is?

We will collect a controlled annotation set. The same news sentences are labeled under three viewing conditions: the sentence alone, the sentence with its structural "ancestors" (headline, lede, and the speaker it is quoted from), and the full article. We then train a hierarchical document encoder, a model that represents text at several levels such as sentence, paragraph, quote, and article, to predict the full spread of labels each reader group gives, not a single label. If structure matters, we gain two things. One is a way to separate disagreement that context resolves from disagreement that reflects genuine differences in perspective. The other is evidence that sentence-level bias datasets carry an artifact of how they were annotated.

Core Claim

We claim that disagreement over sentence-level bias labels in news has a structure tied to the sentence's position in the article. In particular, it should be concentrated in quoted material and shift systematically with the context shown. Two kinds of disagreement should be separable. Some disagreement shrinks when the headline, lede, and attribution are visible; this is the "item ambiguity" component, in the sense of Word-Sense Distinguishability and Inter-Coder Agreement, which separates observer bias from categories that cannot be told apart. The rest persists across reader ideologies; this is the perspective component. A model that encodes the article as an explicit tree, with quotes and speakers as their own level, should predict both the condition-specific label distributions and the residual gap between ideological groups better than a flat model given the same tokens.

Writing $p(y \mid s, C, g)$ for the distribution of labels $y \in$ {neutral, informational bias, lexical bias} given sentence $s$, visible context $C$, and annotator group $g$, the two quantities we estimate are as follows:

$$ \Delta_{\text{context}}(s) = H\big(p(y \mid s, \emptyset)\big) - H\big(p(y \mid s, C_{\text{full}})\big), \qquad D_{\text{persp}}(s) = \mathrm{JS}\big(p(y \mid s, C_{\text{full}}, g_1),\, p(y \mid s, C_{\text{full}}, g_2)\big) $$

Here $H$ is entropy, so $\Delta_{\text{context}}$ measures how much disagreement full context removes. JS is Jensen–Shannon divergence, so $D_{\text{persp}}$ measures how far apart two reader groups remain once context is fixed.

Several findings motivate this. BASIL found that over 40% of bias annotations sit inside quotes. It also found that informational bias, meaning selective presentation of facts, spreads evenly through the article, while loaded wording clusters near the top (In Plain Sight: Media Bias Through the Lens of Factual Reporting). Lim et al. found that annotators with prior knowledge of the event agreed more, which is a direct hint that context reduces disagreement. LimitBias! showed that workers judge statements they agree with as neutral, which is the perspective component we want to isolate.

Research Program

Annotation study. We draw roughly 60 articles from 20 BASIL events, covering the same event from Fox News, the New York Times, and HuffPost. That gives about 1,600 sentences. Each sentence is labeled by about ten annotators in each of the three viewing conditions. The design is between-subjects per sentence, so no annotator sees the same sentence twice. Annotators self-report political lean and whether they already knew about the event. We use descriptive instructions, deliberately unlike the prescriptive adjudication of the FIGNEWS teams (GroningenAnnotatesGaza, Eagles). Those teams pushed agreement up through adjudication, reaching Krippendorff's alpha of 0.420 and Cohen's kappa of 55.5. We want to measure divergence, not suppress it. BASIL's existing consensus labels give us a fixed reference point for the same sentences.

Inducing structure in unmarked news. News text lacks the section headers that TreeJC, HiStruct+, and HIBRIDS depend on. HiStruct+ itself gained only marginally on CNN/DailyMail, which is a warning against importing section-based schemas into news. We therefore build a news-specific tree with the following levels: article root, headline, lede, body paragraphs, and, beneath paragraphs, attributed-speaker nodes that own their quote spans. Quote spans and speakers are found with a quote-attribution tagger. We validate it against BASIL's direct and indirect attribution annotations. Building on TreeJC's finding that removing ancestor headers hurts badly, each sentence is paired with its ancestor chain of headline, lede, and speaker.

Model. A RoBERTa-base sentence encoder feeds an inter-sentence Transformer. Attention between sentences receives learned biases indexed by tree path length and level difference, following HIBRIDS. The output layer predicts the label distribution for each reader group, conditioned on a group embedding. Following Fleisig et al. and GroupAnno, we condition on groups, but we never use annotator IDs at inference. A separate input specifies the viewing condition, so one model can predict what an isolated-sentence annotator versus a full-article reader would say.

Analysis. We estimate $\Delta_{\text{context}}$ and $D_{\text{persp}}$ for each sentence directly from human labels. We then break both down by structural position (quote versus narrative, lede versus body) and by bias type. This analysis stands on its own regardless of how the models perform.

Baselines

The naive reference point predicts, for every sentence, the marginal label distribution of its viewing condition. A second naive reference is a sentence-only RoBERTa-base trained on soft labels. On BASIL's consensus labels, sentence RoBERTa scores 42.13 F1 under the event split (MultiCTX) and 42.16 F1 under the story split (Context in Informational Bias Detection).

The strongest existing method for sentence-level informational bias is MultiCTX. It combines contrastive sentence embeddings with a graph of sentences linked by coreference, discourse markers, and similarity, across the article and across outlets covering the same event. It reports 46.08 F1 on BASIL's event split, above EvCIM's 44.10. It is the strongest competitor because it already uses rich multi-level context, just not the article's physical tree. We will retrain it on our soft labels with the same backbone and the same event-wise splits. Our model will not beat 46.08 by much on consensus F1, and that is not the claim. On consensus F1 we only require that it does not fall below EvCIM's 44.10. The real test is distribution prediction, where no reported result was found on Lacuna for soft-label or group-conditioned bias prediction on news.

The flat structural competitor follows Buchmann et al., which added structural tokens and depth/type embeddings to long-document Transformers and gained 14.04 F1 on QASPER evidence selection. We give a flat full-article Longformer the same induced quote, speaker, and lede markers as tokens. This isolates whether a hierarchical encoder adds anything beyond structural information alone.

Our key ablations remove the speaker/quote level, keeping only headline, lede, and paragraph, and remove the tree attention biases. All models share RoBERTa-base-scale backbones, the same 20-trial hyperparameter search, and five seeds. Beating a baseline means a lower held-out Jensen–Shannon divergence to the human distribution, with a paired bootstrap 95% interval that excludes zero, overall and specifically on quoted sentences.

Why not just…?

Why not just add hand features such as "is in quote" and "paragraph index" to a sentence classifier? This is a real threat. If a logistic regression on sentence embeddings plus a few position features predicts the distributions as well as the tree model, then the finding is that position explains disagreement, and the hierarchical encoder is unnecessary. We run exactly this baseline and will report that outcome if it occurs. The descriptive analysis of $\Delta_{\text{context}}$ and $D_{\text{persp}}$ is the core result either way. The tree model only earns its place if it captures interactions such as which speaker is quoted relative to the headline's framing.

Why not just give a long-context model the whole article? Context in Informational Bias Detection found that neither a full-article BiLSTM nor windowed neighboring sentences meaningfully beat the sentence baseline. That constrains naive full-article context. The Buchmann-style flat model with structural tokens is our fair version of this alternative. We also include persona-prompted large-model zero-shot prediction as a check. DEM-MOE (Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives) reports that persona prompting struggled on nuanced tasks, but that was on short texts, so we test it directly here.

Why not just collect more annotators and aggregate with a truth-inference model? Aggregation assumes there is one latent truth. GroupAnno's gain over majority vote was small (0.968 versus 0.955 F1), and it came from modeling systematic group effects, not from removing them. More importantly, aggregation cannot tell whether disagreement disappears when context is shown. Only the three-condition design measures that.

Why not just use DEM-MOE or Fleisig et al. on sentences? Those models condition on who the annotator is, but they treat the input as an isolated utterance. We include a DEM-MOE-style sentence-only group-conditioned model as a baseline. If it matches our model on the full-article condition, structure adds nothing to perspective modeling.

Decisive Experiments

The first check is purely human. Does disagreement entropy drop significantly from the isolated condition to the full-article condition? Is the drop larger for quoted sentences than for narrative ones? If $\Delta_{\text{context}}$ is near zero everywhere, the structural half of the proposal fails and we report that. That result would itself rebut the assumption that isolated-sentence annotation is harmless.

The second check is whether $D_{\text{persp}}$ remains above a permutation-test null after context is fixed. This would show that some disagreement is ideological rather than an artifact of what annotators could see.

The third check compares the tree encoder against the flat structural-token model and the hand-feature model on Jensen–Shannon divergence, stratified by quote status and position in the article.

The fourth check tests transfer of annotation artifacts. A model trained on isolated-condition labels, the way BASIL-style corpora are built, is evaluated against full-article human distributions. A large gap would indicate that existing sentence bias benchmarks measure a reading situation no real reader is in.

As a secondary sanity check, we report consensus F1 on BASIL's event split against MultiCTX's 46.08.