A Multilingual Similarity Dataset for News Article Frame

Xi Chen, Mattia Samory, Scott Hale, David Jurgens, Przemyslaw A. Grabowicz

In communication studies, a fundamental distinction exists between agenda-setting (deciding which stories are important) and framing (deciding how those stories are presented). While computational linguistics has made significant strides in clustering news by topic, the nuance of framing—encompassing perspective, tone, and narrative structure—has largely remained a "black box" for large-scale analysis. Existing datasets are frequently limited to single languages, focus exclusively on headlines, or utilize coarse categories that fail to generalize across diverse political and cultural landscapes.

To bridge this gap, researchers have released the most extensive multilingual dataset to date for measuring news article similarity. Featuring over 26,000 labeled pairs across 10 languages, this work introduces a specific "FRAME" aspect to quantify the narrative delivery of news, enabling new methodologies for measuring media bias and global news synchrony.

Methodology: Human-in-the-Loop Active Learning

Building a high-quality similarity dataset is challenging because the vast majority of random article pairs are entirely unrelated. To overcome this, the authors employed a human-in-the-loop active learning framework:

  1. Sampling: A logistic regression model was used to identify "potentially similar" pairs from a pool of 60 million articles. This model filtered candidates based on shared named entities, word counts, and publication time proximity.
  2. Annotation: 25 trained annotators evaluated the resulting pairs based on an eight-aspect codebook: Geography (GEO), Entities (ENT), Time (TIME), Narrative (NAR), Overall Similarity (OVERALL), Style (STYLE), Tone (TONE), and the novel Frame (FRAME).
  3. Modeling: The team developed a computationally efficient transformer model to infer similarity. The model achieved a Pearson correlation of 0.8 with human labels, a level of performance comparable to the average human annotator.

Dataset Composition

The dataset is derived from articles published between January and June 2020, sourced via Media Cloud.

  • Total Scale: 26,555 labeled news article pairs.
  • Language Diversity: 10 languages (English, Spanish, Russian, German, French, Arabic, Italian, Turkish, Polish, and Mandarin Chinese), covering approximately 76% of the global population.
  • Cross-lingual Analysis: The set includes 4,214 pairs where the articles are in different languages (e.g., a German article paired with an English one).
  • Frame Focus: A specialized subset of 1,522 pairs is specifically labeled for the "FRAME" aspect, providing a granular look at narrative variance.

"How" vs. "What"

The research confirms that articles can cover the exact same event (high "Overall" similarity) while employing radically different frames. For example, a COVID-19 lockdown might be framed as a necessary public health intervention in one outlet, while another frames it as an infringement on individual or religious liberties.

The Synchrony Network

By analyzing millions of pairs, the authors mapped a "global news backbone." Their findings suggest that news synchrony—the phenomenon of different countries reporting on the same topics at the same time—is predicted more accurately by trade volume and shared language than by geographic proximity.

Quantifying Media Bias

The study found a measurable correlation between an outlet's political leaning and the framing of its content. As the political distance between two outlets increases, the similarity of their frames and tones decreases in a predictable fashion. Additionally, using a tool called Riveter, the authors found that Joe Biden was consistently portrayed with higher "power scores" (associated with institutional prestige) compared to Donald Trump, who was more frequently associated with coercive or defensive language across the sampled media.

Results and Reliability

The dataset demonstrates high inter-rater reliability, particularly regarding "OVERALL" similarity, with a Krippendorff’s alpha of 0.77. Furthermore, the study identified that high internet penetration is the strongest predictor of news diversity within a country, suggesting that digital access allows media ecosystems to cater to a broader range of interests and motivations.

Limitations and Future Work

While the dataset is a landmark contribution, the authors note that the FRAME aspect was only labeled for a subset of 1,522 pairs, as framing can only be meaningfully judged when articles target the same subject. Additionally, while 10 languages represent a massive step forward, many regional dialects remain unrepresented.

Future applications of this framework include analyzing long-term trends in media bias during major geopolitical shifts and identifying "news deserts"—regions where significant local events are under-reported by the global media compared to Western-centric stories.

Summary updated

BibTeX
@inproceedings{chen2024multilingual,
  author = {Xi Chen and Mattia Samory and Scott Hale and David Jurgens and Przemyslaw A. Grabowicz},
  title = {{A Multilingual Similarity Dataset for News Article Frame}},
  booktitle = {ICWSM},
  year = {2024},
  doi = {10.1609/icwsm.v18i1.31435},
}
Figure