TL;DR
This paper introduces MediaRank, a novel system that computationally ranks over 50,000 online news sources globally. By analyzing a vast dataset of news articles and tweets, MediaRank leverages four key signals—peer reputation, reporting bias/breadth, financial pressure, and popularity—to provide an interpretable and validated measure of news quality, addressing the growing distrust in traditional media.
What Problem Does This Paper Address?
In an era marked by increasing distrust in traditional news media and the proliferation of "fake news," it's become challenging for individuals to identify reliable sources of information. While NLP methods can detect unreliable articles and third-party fact-checkers exist, these approaches are often too slow or brittle for the fast-paced news cycle. The authors argue that assessing news quality at the source level is more effective and scalable. However, existing expert-driven rankings, while valuable, struggle with comprehensive coverage due to the sheer volume of news generated daily. This paper aims to bridge this gap by developing an automated, large-scale system for ranking news sources based on established journalistic quality indicators.
Methods
MediaRank employs a multi-faceted approach to assess news source quality, combining several novel computational methods:
- Peer Reputation: This is measured using a PageRank algorithm applied to a news citation graph. The idea is that reliable news sources are more likely to be cited by other reputable sources, acting as endorsements.
- Reporting Bias and Breadth: Bias is quantified by analyzing sentiment differences towards political parties across a large corpus of news articles. Breadth is measured by the number of unique entities (e.g., celebrities) mentioned in articles, aiming to capture diverse reporting.
- Bottomline Pressure: This is a novel contribution, with two metrics proposed:
- Social Bot Detection: A classifier is trained to identify social media accounts used to artificially boost website traffic.
- Advertisement Aggressiveness: The number and placement of ads on news pages are analyzed to gauge financial pressure.
- Popularity: This is measured using social media engagement (e.g., tweet mentions) and Alexa rank, reflecting how widely a source is consumed by readers and other sources.
These individual signals are then combined into a final quality score.
Datasets
- News Webpages and Tweets: The system collects approximately one million news webpages and two million related tweets daily.
- News Citation Graph: Constructed from articles collected between September 24, 2017, and February 16, 2018 (23,371,264 articles), containing over 64 million citations.
- Social Bot Training Data: Two datasets are used:
- Botometer: A combination of four public social bot datasets.
- Removed Accounts: Accounts deleted by Twitter, indicating potential bot activity.
- External Bias Datasets:
- AllSides: Provides political bias labels for 222 raw news sources.
- MediaBiasFactCheck (MBFC): Contains bias labels for 1040 news sources.
- Expert News Rankings: The paper validates MediaRank against 35 published news rankings across various languages and countries.
Key Insights and Ideas
- Citations as Endorsements: The paper posits that news sources citing each other is a strong indicator of peer reputation, a concept that can be computationally leveraged via graph analysis.
- Bias and "Echo Chambers": Interestingly, the analysis of news citation embeddings suggests that news sources cite each other despite different political views, contradicting the notion of strict "echo chambers" in news consumption.
- Political Leanings of Top Sources: In four major English-speaking countries (US, UK, Canada, Australia), the highest-ranking news sources disproportionately favor liberal parties, even when the majority of news sources exhibit conservative slants.
- Distinguishing URL vs. Citation Graphs: The authors highlight that traditional website ranking algorithms (like PageRank on URL graphs) can be misled by "sponsored articles" and uninformative links, whereas citation graphs provide a more robust measure of journalistic reputation.
- Bottomline Pressure as a Quality Indicator: The paper introduces novel metrics for financial pressure (social bots, ad aggressiveness), recognizing it as a significant factor influencing news quality.
Key Results
- MediaRank provides interpretable quality rankings for over 50,000 news sources.
- MediaRank scores show a positive correlation with 34 out of 35 expert news rankings, with a mean Spearman coefficient of 0.58.
- The system successfully identifies significant left and right-leaning news sources based on sentiment analysis.
- The social bot classifier achieves high performance, with XGBoost outperforming SVM and logistic regression on both bot datasets.
- The paper demonstrates that higher-ranked news sources tend to publish more articles, have more citations, and receive more citations from others.
Limitations and Future Work
- Language Support: The authors acknowledge the need for improved non-English language support, particularly for political bias measurement.
- NLP Depth: Deeper NLP analysis of articles to verify factual claims is a long-term goal.
- Data Release: The paper mentions making the analysis and rankings publicly available at www.media-rank.com, which is a significant contribution.
- Future Visualization: Work is ongoing to improve visualization techniques for news analysis.
Summary updated