Abstract
In this research, we continuously collect data from the RSS feeds of traditional news sources. We apply several pre-trained implementations of named entity recognition (NER) tools, quantifying the success of each implementation. We also perform sentiment analysis of each news article at the document, paragraph and sentence level, with the goal of creating a corpus of tagged news articles that is made available to the public through a web interface. Finally, we show how the data in this corpus could be used to identify bias in news reporting.
Introduction
Identifying systemic bias in news reporting is a complex linguistic challenge. While audiences rely on media for objective context, traditional resources like the New York Times Annotated Corpus or GDELT are often static, costly, or prioritize volume over granular detail. The "Quantum Criticism" project addresses these gaps by introducing a dynamic, multi-layered news corpus that tracks sentiment and named entities in real-time across major media outlets.
A core argument of this work is that traditional sentiment analysis—which often averages the tone of an entire article—misses subtle nuances. A report on a tragedy may have a negative overall tone, yet specific mentions of a public figure within that report might be positive. By analyzing sentiment at the "quantum" level (sentences and paragraphs), this framework links specific entities (people, places, organizations) to high-resolution sentiment scores.
Methodology
The researchers developed a proprietary pipeline to ingest, process, and store news data systematically.
The pipeline consists of several key stages:
- Ingestion: Custom scrapers monitor RSS feeds from five major sources—The Atlantic, BBC, Fox News, NYT, and Slate—every two hours.
- Sentiment Analysis: Two distinct tools are utilized: VADER (rule-based, sensitive to social media nuances) and CoreNLP (neural-based).
- Named Entity Recognition (NER): Eight different models across BERT, spaCy, and CoreNLP identify three primary categories: PERSON, LOCATION, and ORGANIZATION.
- Entity Resolution: A custom algorithm links partial names (e.g., "Pelosi") or acronyms (e.g., "FBI") to their full identities ("Nancy Pelosi" or "Federal Bureau of Investigation") across the corpus.
- Quantum Analysis: Sentiment is calculated for every sentence and paragraph, allowing researchers to associate an entity with the specific "affect" of the text immediately surrounding it.
The Quantum Criticism Corpus
The primary output is a growing dataset that contained over 105,000 articles as of late 2019. To validate the system, the researchers used the LDC New York Times Annotated Corpus as a baseline and Ballotpedia to create a "gold standard" list of the 538 members of the U.S. Congress.
As shown in the growth trends, outlets like Fox News and the New York Times contribute the highest volume of data, while others like The Atlantic provide a different geographic variety despite fewer articles.
Key Insights and Results
The project yielded several significant findings regarding media analysis and tool performance:
- Automated vs. Manual Tagging: Validation revealed that the "manually tagged" NYT corpus often under-reported entities. Automated models identified many valid entities missed by human annotators, suggesting that modern NLP tools may now surpass older manual benchmarks in comprehensiveness.
- Sentiment Hierarchies: In left-leaning news, a specific pattern emerged where sentiment was lowest at the article level but significantly higher in specific sentences mentioning favored politicians. This hierarchy shifts during seminal events; for instance, during a retirement announcement, article-level sentiment may spike higher than sentence-level sentiment.
- NER Performance: The system achieved 96.9% accuracy in identifying "PERSON" entities across eight NER tools. However, high-profile individuals are occasionally misidentified as "ORGANIZATIONS" (2.98%–5.06% rate) due to their strong association with institutions (e.g., "Speaker of the House").
- Entity Resolution Success: The system successfully resolved "Nancy Pelosi" in over 400 different variations, including titles and misspellings.
Limitations and Future Directions
While robust, the framework faces ongoing challenges. NER models still produce "spurious" labels—such as BERT tagging common nouns in headlines as entities. Currently, the corpus is primarily focused on five American and British sources.
Future work aims to integrate social media data (e.g., Twitter) to analyze the feedback loop between professional news and public reaction. Additionally, the team plans to incorporate frameworks like CAMEO or COPDAB to code event data more effectively, enabling deeper cross-national comparisons of event reporting.
Summary updated