Framing in the Presence of Supporting Data: A Case Study in U.S. Economic News

Alexandria Leto, Elliot Pickens, Coen D. Needell, David Rothschild, Maria Leonor PachecoACL 2024 · DOI

Introduction

Media outlets possess significant influence over public perception through the twin mechanisms of selection (choosing which facts to report) and framing (applying a narrative lens to those facts). While studying media bias has traditionally been hampered by subjectivity, economic news provides a unique opportunity for objective analysis. Economic indicators—such as GDP, unemployment rates, and stock market indices—offer a "ground truth" of numerical values against which reporting can be measured.

This paper introduces a computational framework designed to quantify media framing by anchoring news reports to these objective data points. By analyzing eight years of U.S. economic news, the researchers demonstrate how a relational modeling approach can automatically identify which economic indicators a publisher selects and how they "spin" those numbers to shape a specific narrative.

Methodology: Relational Modeling of Economic Frames

The authors propose a multi-level computational framework that treats frame prediction as a set of interdependent tasks rather than isolated classification problems.

Figure 1: Example of the multi-level annotation task, showing how specific quantities (e.g., stock market declines) are linked to overall article-level framing and economic conditions.

The framework operates across several layers:

  1. Article-Level Classification: The model identifies the general topic (e.g., macro-economic vs. firm-specific) and the overall sentiment, determining if the economy is portrayed as "good" or "poor" and whether it is "getting better" or "worse."
  2. Quantity-Level Classification: Every numerical value within an article is analyzed to determine if it refers to a specific economic indicator (e.g., "The S&P 500 fell 3%") and is assigned a polarity (positive, negative, or neutral).
  3. Probabilistic Soft Logic (PSL): To ensure consistency, the model uses PSL to enforce logical constraints. For instance, if an article highlights multiple negative specific quantities, the overall article-level sentiment is logically constrained to reflect that negativity.
  4. Domain-Adaptive Pre-training (DAPT): A RoBERTa base model was fine-tuned on a massive unlabeled corpus of economic news to capture the nuances of financial language before being trained on the labeled dataset.

Data and Annotation

The study utilizes a substantial new dataset comprising 199,066 articles from the landing pages of six major U.S. publishers (The New York Times, Wall Street Journal, Washington Post, Fox News, Breitbart, and HuffPost) spanning 2015 to 2023.

Figure 2: Distribution of article types and economic labels within the annotated dataset, highlighting the prevalence of industry and macro-economic reporting.

To train and validate the models, a subset of 1,171 articles and 2,414 quantities were annotated by experts. This internal data was then compared against external "ground truth" data from the Bureau of Labor Statistics (BLS) and the Federal Reserve (FRED).

Key Insights and Results

The research yielded several critical findings regarding the nature of media framing:

  • Selection as Framing: The most powerful editorial tool is often the selection of which data to present. Outlets may ignore positive indicators to focus on negative ones (like rising energy prices) to maintain a specific narrative.
  • The Power of Joint Inference: The Relational (PSL) model significantly outperformed independent classifiers and LLM baselines, achieving a Macro F1 of 0.717 for economic conditions and 0.849 for macro-indicators. Modeling the relationship between individual data points and the overall article tone proved essential.
  • Negativity Bias and Publisher "Personalities": The framework revealed that some outlets, such as The New York Times, maintained negative framing even during periods of economic recovery. In contrast, the Wall Street Journal showed more "malleable" framing that tracked more closely with actual data shifts.
  • Model Efficiency: Specialized fine-tuning (RoBERTa + DAPT) outperformed large language models like Mistral-7B in zero-shot and few-shot settings, highlighting the value of domain-specific training for complex classification tasks.

Conclusion and Future Directions

While the current study focuses on the economy, the framework is designed for expansion into any field with objective supporting data, such as crime statistics or climate change. Future work will address data skew in less-frequent indicators and explore the use of LLMs to generate "silver-standard" labels to scale the training process. Ultimately, this research provides a robust tool for holding media outlets accountable by comparing their narratives to the objective reality of the data they report.

Summary updated

BibTeX
@inproceedings{leto2024framing,
  author = {Alexandria Leto and Elliot Pickens and Coen D. Needell and David Rothschild and Maria Leonor Pacheco},
  title = {{Framing in the Presence of Supporting Data: A Case Study in U.S. Economic News}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.24},
}
Figure