SciLens News Platform: A System for Real-Time Evaluation of News Articles

Angelika Romanou, Panayiotis Smeros, Carlos Castillo, Karl AbererarXiv 2020 · DOI · arXiv

This blog-style summary provides a technical overview of the SciLens News Platform, as presented in the paper "SciLens News Platform: A System for Real-Time Evaluation of News Articles."

TL;DR

SciLens is a real-time platform that bridges the gap between traditional expert fact-checking and automated computational journalism. By fusing content analysis, social media signals, and expert reviews, it provides a multi-dimensional quality score to help users navigate scientific misinformation, particularly during high-stakes events like the COVID-19 pandemic.

What Problem Does This Paper Address?

In an era of "information inflation," the decentralization of news through social media has led to a surge in misinformation. Non-experts often struggle to distinguish credible scientific reporting from misleading content, which may stem from malicious intent or simply a lack of domain expertise.

Traditional manual fact-checking (e.g., Snopes) is highly reliable but cannot scale to the thousands of articles produced daily. Conversely, purely computational methods often lack the nuance of human judgment. The authors address this by creating a hybrid system that provides real-time, automated quality indicators alongside expert annotations to foster a better consensus on news quality.

Methods

The SciLens platform utilizes a three-pronged approach to evaluate news quality:

  1. Automated Quality Indicators:
    • Content: The system analyzes the article's body and title for clickbaitness, subjectivity, and readability.
  2. News Context: It examines the "journalistic foundations" by categorizing references. High-quality articles are expected to have more links to academic repositories and peer-reviewed journals rather than just internal links to the same news outlet.
  3. Social Media Context: It measures reach (popularity) and stance (whether social media users are supporting the claims or expressing doubt/contradiction).
  4. Expert Reviews: Experts can annotate articles based on seven criteria: factual accuracy, scientific understanding, logic/reasoning, precision/clarity, source quality, fairness, and clickbaitness. These are aggregated into a weighted, time-sensitive score.

  5. System Architecture: As shown in the architecture diagram below, the system uses a streaming pipeline (Datastreamer API) to ingest data in real-time. It employs a hybrid storage strategy—an RDBMS for real-time operations and a distributed Hadoop/Spark environment for historical data warehousing and machine learning model training.

Figure 2: SciLens News Platform architecture. First, a stream- ing pipeline acts as the entry point of data collection. Then, a data layer, comprised by an RDBMS and a Distributed Stor- age, stores the incoming data. Lastly, the analytics layer man- ages the data, trains the Machine Learning models and serves the extracted indicators to the web application.

Datasets

The authors evaluated the platform using a dataset focused on the COVID-19 pandemic collected between January 15 and March 15, 2020.

  • Outlets: A shortlist of 45 mainstream news outlets.
  • Ground Truth: Quality rankings for these outlets were imported from the American Council on Science and Health to serve as a baseline for comparing high-quality vs. low-quality sources.

Key Insights and Ideas

  • The Quantity-Quality Trade-off: During the early stages of the pandemic, low-quality outlets significantly increased their publication volume compared to high-quality outlets. They appear driven by "breaking news" cycles, whereas high-quality outlets remain more conservative and evidence-focused.
  • The Reach Paradox: Low-quality outlets often achieve a wider distribution of social media reactions. This suggests that sensationalism or high-frequency posting effectively captures attention, even if the underlying scientific foundation is weak.
  • Evidence Seeking: There is a clear divergence in how outlets use references. High-quality outlets consistently show a higher ratio of scientific references (linking to academic journals) compared to their low-quality counterparts.
  • Consensus Building: The authors found that providing these heterogeneous indicators helps non-expert users reach a more accurate consensus on an article's quality, bringing their assessments closer to those of domain experts.

Key Results

The platform successfully operates at scale, handling thousands of articles daily. The user interface (shown below) provides a comprehensive dashboard for both single-article assessment and broader topic insights.

Figure 3: User interface of the SciLens News Platform.

In the COVID-19 use case, the system quantitatively demonstrated that:

  • Low-quality outlets have a wider distribution of social media reactions but a lower number of scientific references.
  • High-quality outlets show the inverse: fewer social media "hits" but much stronger scientific grounding.

Limitations and Future Work

While the platform is robust, the paper notes that the expert review component is inherently less scalable than the automated indicators. The authors suggest that the platform's current strength lies in scientific news, where reference patterns are more distinct. Future work could involve refining the "stance" detection algorithms to better handle the nuances of social media sarcasm or complex scientific debates. Additionally, expanding the platform to handle non-English news sources remains a significant frontier for global misinformation tracking.

Summary updated

Abstract

We demonstrate the SciLens News Platform, a novel system for evaluating the quality of news articles. The SciLens News Platform automatically collects contextual information about news articles in real-time and provides quality indicators about their validity and trustworthiness. These quality indicators derive from i) social media discussions regarding news articles, showcasing the reach and stance towards these articles, and ii) their content and their referenced sources, showcasing the journalistic foundations of these articles. Furthermore, the platform enables domain-experts to review articles and rate the quality of news sources. This augmented view of news articles, which combines automatically extracted indicators and domain-expert reviews, has provably helped the platform users to have a better consensus about the quality of the underlying articles. The platform is built in a distributed and robust fashion and runs operationally handling daily thousands of news articles. We evaluate the SciLens News Platform on the emerging topic of COVID-19 where we highlight the discrepancies between low and high-quality news outlets based on three axes, namely their newsroom activity, evidence seeking and social engagement. A live demonstration of the platform can be found here: http://scilens.epfl.ch.
BibTeX
@misc{romanou2020scilens,
  author = {Angelika Romanou and Panayiotis Smeros and Carlos Castillo and Karl Aberer},
  title = {{SciLens News Platform: A System for Real-Time Evaluation of News Articles}},
  year = {2020},
  eprint = {2008.12039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.12039},
}
Figure