NeMig -- A Bilingual News Collection and Knowledge Graph about Migration

Andreea Iana, Mehwish Alam, Alexander Grote, Nevena Nikolajevic, Katharina Ludwig, Philipp Müller, Christof Weinhardt, Heiko PaulheimarXiv 2023 · arXiv

This paper introduces NeMig, a specialized bilingual news dataset designed to move the field of news recommendation beyond simple click-prediction accuracy and toward understanding complex societal effects like filter bubbles and political polarization.

TL;DR

NeMig is a new bilingual (German and English) dataset focused on the controversial topic of migration, combining news articles with rich annotations (sentiment, political leaning, subtopics) and detailed user socio-demographic and political data. By providing domain-specific knowledge graphs and explicit user feedback, it enables researchers to analyze how recommendation algorithms influence opinion formation and reinforce biases in a way that standard, general-purpose datasets cannot.

What Problem Does This Paper Address?

Most modern news recommendation research relies on massive, general-purpose datasets (like Microsoft’s MIND) that focus primarily on "click-through rate" (CTR) optimization. While effective for engagement, this approach ignores the fundamental role media plays in shaping public worldview, especially regarding sensitive or polarizing topics.

Existing resources typically suffer from two gaps:

  1. Recommendation datasets are usually monolingual, cover "soft" topics (sports, entertainment), and lack deep information about the users' own political leanings or the articles' ideological slants.
  2. Media analysis datasets (used for bias or fake news detection) often have rich annotations but lack the user interaction data required to train or evaluate recommendation algorithms.

NeMig bridges this gap by providing a "thick" dataset centered on a single, highly polarizing topic—migration—allowing for interdisciplinary research into how algorithms interact with human political psychology.

Methods

The authors developed a comprehensive pipeline for data collection, annotation, and knowledge graph construction:

  • Data Collection: They crawled over 18,000 articles from 40 German and 45 US news outlets, carefully selected to span the entire political spectrum.
  • Multi-faceted Annotation:
    • Sentiment: Articles were classified as positive, neutral, or negative using a multilingual XLM-R model.
  • Sub-topics: Neural topic modeling identified specific themes (e.g., "border control," "Russian invasion of Ukraine").
  • Named Entity Recognition (NER) & Linking: Entities (people, places, orgs) were extracted and disambiguated by linking them to Wikidata QIDs.
  • Knowledge Graph (KG) Construction: The authors built NeMigKG, representing articles, entities, and metadata as nodes. They created four variants of the KG, including versions enriched with up to 2-hop neighbors from Wikidata to provide external context.
  • User Study: Unlike datasets that scrape anonymous logs, the authors conducted controlled online studies with ~3,000 users in both Germany and the US. They collected explicit feedback (rating articles) alongside detailed demographics, political attitudes, and personality traits.
Corpora annotation and knowledge graph construction pipeline

Datasets

The primary contribution is the NeMig dataset itself, which includes:

  • German Corpus: 7,346 filtered articles (2019–2021).
  • English Corpus: 10,814 filtered articles (2021–2022).
  • User Data: Demographics and political profiles for ~6,000 total users.
  • NeMigKG: A domain-specific knowledge graph available in base, entity-only, and Wikidata-enriched versions.

Key Insights and Ideas

  • Knowledge-Awareness Wins on Small Data: In their benchmarks, knowledge-aware models like DKN and CAUM generally outperformed standard neural recommenders. This suggests that when training data is relatively small (thousands of articles rather than millions), injecting external knowledge via KGs is a powerful way to improve representations.
  • The "1-Hop" Sweet Spot: When enriching KGs with external data from Wikidata, the authors found that adding 1-hop neighbors improved diversity and performance, but going to 2-hops often "diluted" the domain-specific signal, leading to worse results.
  • Diversity vs. Personalization: The study confirmed that users tend to prefer news that aligns with their existing topical interests and political leanings rather than their sentiment preferences.
  • Secondary Goals Don't Kill Accuracy: Experiments with SentiDebias showed that it's possible to optimize for sentiment diversity without significantly sacrificing standard recommendation accuracy.

Key Results

The authors benchmarked several state-of-the-art models (NRMS, NAML, DKN, etc.) on NeMig:

  • Content Personalization: DKN and CAUM achieved the best results, with DKN reaching an AUC of 58.73 on the German dataset.
  • Diversity: SentiDebias successfully provided the most sentiment-diverse recommendations. NAML was the most effective at providing diversity across topical categories.
  • Political Bias: Interestingly, all benchmarked recommenders performed nearly identically regarding political diversification, likely because none were explicitly designed to counter political "filter bubbles."
Fig. 2. Distribution of outlets and news over political orientation classes.

Limitations and Future Work

  • Scale: While "thick" in features, the user base (3K per language) is small compared to industry-scale datasets. The authors suggest using this real-world data as a seed to generate larger synthetic user datasets.
  • Copyright Constraints: Due to legal restrictions, the full body text of the news articles is only available upon request for research, though all other features and metadata are freely available on GitHub.
  • Future Directions: The authors aim to use NeMig to study whether recommendation algorithms actively amplify existing user biases over time, rather than just reflecting them.

Summary updated

Abstract

News recommendation plays a critical role in shaping the public's worldviews through the way in which it filters and disseminates information about different topics. Given the crucial impact that media plays in opinion formation, especially for sensitive topics, understanding the effects of personalized recommendation beyond accuracy has become essential in today's digital society. In this work, we present NeMig, a bilingual news collection on the topic of migration, and corresponding rich user data. In comparison to existing news recommendation datasets, which comprise a large variety of monolingual news, NeMig covers articles on a single controversial topic, published in both Germany and the US. We annotate the sentiment polarization of the articles and the political leanings of the media outlets, in addition to extracting subtopics and named entities disambiguated through Wikidata. These features can be used to analyze the effects of algorithmic news curation beyond accuracy-based performance, such as recommender biases and the creation of filter bubbles. We construct domain-specific knowledge graphs from the news text and metadata, thus encoding knowledge-level connections between articles. Importantly, while existing datasets include only click behavior, we collect user socio-demographic and political information in addition to explicit click feedback. We demonstrate the utility of NeMig through experiments on the tasks of news recommenders benchmarking, analysis of biases in recommenders, and news trends analysis. NeMig aims to provide a useful resource for the news recommendation community and to foster interdisciplinary research into the multidimensional effects of algorithmic news curation.
BibTeX
@misc{iana2023nemig,
  author = {Andreea Iana and Mehwish Alam and Alexander Grote and Nevena Nikolajevic and Katharina Ludwig and Philipp Müller and Christof Weinhardt and Heiko Paulheim},
  title = {{NeMig -- A Bilingual News Collection and Knowledge Graph about Migration}},
  year = {2023},
  eprint = {2309.00550},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00550},
}
Figure