Abstract: We describe work in progress on detecting and understanding the moral biases of news sources by combining framing theory with natural language processing. First we draw connections between issue-specific frames and moral frames that apply to all issues. Then we analyze the connection between moral frame presence and news source political leaning. We develop and test a simple classification model for detecting the presence of a moral frame, highlighting the need for more sophisticated models. We also discuss some of the annotation and frame detection challenges that can inform future research in this area.
Introduction
While "fake news" often dominates public discourse, news framing—the subtle highlighting of specific aspects of an event to promote a particular moral evaluation—is a more pervasive force in shaping public opinion. For instance, a protest can be framed as a "fight for justice" or "criminal subversion." This research addresses the challenge of automatically identifying these moral biases using Moral Foundations Theory (MFT), which categorizes human morality into five pillars: Care/Harm, Fairness/Cheating, Loyalty/Betrayal, Authority/Subversion, and Sanctity/Degradation.
Methods and Data
The study utilized a dataset of 400 news articles covering controversial topics such as immigration, smoking, same-sex marriage, and the Baltimore racial unrest. The researchers developed a two-stage annotation protocol for human labels at both sentence and article levels.
To detect these frames automatically, two approaches were tested:
- Keyword Matching: Utilizing a dictionary of words associated with MFT.
- Support Vector Machines (SVM): Training a classifier using
word2vecembeddings to capture semantic meaning.
Additionally, a domain expert mapped issue-specific frames (e.g., "Police Racism" vs. "Black Criminality") to broader moral foundations to analyze how specific "spin" correlates with universal moral values.
Key Insights and Results
A significant finding was the "presence paradox": while human annotators struggled to agree on whether a moral frame was present in a sentence (due to extreme class imbalance), they were highly consistent in identifying which foundation it was once they agreed a frame existed.
As shown in Figure 1, "Harm" and "Cheating" emerged as the most prevalent moral frames across the dataset, likely because these foundations are valued across the political spectrum. However, moral frames also served as a political "fingerprint." For example, conservative sources utilized the Subversion frame (criminalizing protesters) significantly more, while liberal sources leaned on Fairness and Cheating (focusing on systemic inequality).
In terms of automated detection, binary classification (detecting the presence of a frame) proved difficult, with the SVM achieving a modest F-score of 0.27. However, for multi-class classification (identifying the specific frame among those known to have one), the SVM performed much better with an F-score of 0.68, significantly outperforming the keyword baseline of 0.31. This suggests that semantic context is far more critical than simple "moral keywords."
Limitations and Future Work
The research highlights several challenges, including data scarcity and the inherent subjectivity of moral framing. The low inter-annotator agreement on the presence of frames suggests that identifying morality in text may require more expert training or refined annotation guidelines. Future work will focus on scaling the dataset to leverage transformer-based models and exploring how these moral frames propagate through social media to influence public polarization.
Summary updated