TL;DR
This paper tackles the challenge of automatically identifying "frame-invoking language" in political news, a crucial step for understanding how issues are presented and potentially mitigating framing effects. They developed and tested computational approaches, finding that lexical and theoretical features, combined, achieve performance comparable to human annotators in identifying this language.
What Problem Does This Paper Address?
Political issues are rarely presented neutrally; they are "framed" – defined, linked to other problems, and imbued with moral judgments. This framing significantly influences public perception and action. While the concept of framing is well-studied, there's a lack of clear guidelines for computationally identifying the specific language that invokes these frames. This makes it difficult to study framing effects systematically or to develop tools that help people reflect on and understand framing. The paper aims to bridge this gap by developing and evaluating methods to automatically detect frame-invoking language, particularly in news coverage where framing can be subtle.
Methods
The authors developed a computational approach to identify frame-invoking language. This involved:
- Pilot Study: A crucial first step was a pilot study with 47 undergraduate students. Participants were asked to highlight words and phrases they believed were related to framing in a healthcare article. This qualitative data helped understand how laypeople identify framing and informed feature selection for the computational models.
- Data Collection: News articles were collected from top online news sources.
- Annotation: The collected articles were annotated by multiple annotators (Mechanical Turk workers and university students) who were given similar instructions to the pilot study participants. This resulted in a dataset of annotated words and phrases.
- Feature Engineering: A wide array of linguistic features were engineered to represent words and their contexts. These included:
- Lexical Features: The words themselves, their lemmas, and n-grams (sequences of words).
- Grammatical Features: Parts of speech, sentence structure roles, and named entity types.
- Document Features: Sentence length and TF-IDF scores.
- Theoretical Features: Features derived from linguistic theories of framing, such as imagery, figurativeness, abstractness, and subjective/bias words.
- Classifier Development: Various machine learning classifiers were tested, with Naïve Bayes performing best. The authors then evaluated different combinations of these features to see which were most effective.
Datasets
The paper introduces a newly annotated dataset for identifying frame-invoking language.
- Dataset: 205 political news articles were collected from eight major news sources. A subset of 75 articles was randomly selected for annotation.
- Annotation: Each article was annotated by 3-11 annotators. The final dataset used for training and testing consists of 74 articles with 53,878 total words, and 59,948 annotated words across 507 annotations from 372 annotators. This dataset is made publicly available.
Key Insights and Ideas
- Laypeople's Understanding Matters: The pilot study revealed that laypeople's understanding of framing is crucial for developing tools that support frame reflection. Their insights guided the feature engineering process.
- Framing is More Than Just Opinion: The authors distinguish framing from subjectivity, sentiment, and bias, noting that framing can support the same position through different rhetorical lenses.
- Context is Key, But Not Always Obvious: While words themselves are important, their surrounding context and relationships to other terms significantly influence whether they are perceived as frame-invoking.
- Lexical Features Dominate, But Theory Helps: The best performing classifiers relied heavily on lexical features (the words themselves and their immediate context). However, incorporating theoretical features related to imagery, abstractness, and subjective language also proved valuable, suggesting that while specific words are strong indicators, deeper linguistic properties contribute.
- Grammatical Structure is Less Predictive Than Expected: Surprisingly, grammatical structure features alone did not perform as well as lexical or theoretical features, despite initial hypotheses and prior work on related tasks.
Key Results
- Human-like Performance: The best performing computational classifiers achieved F1 scores comparable to human annotators (around 0.45-0.46).
- High Recall, Lower Precision: The classifiers were more aggressive than humans, achieving significantly higher recall (identifying more frame-invoking language) but lower precision (identifying more non-frame-invoking language as frame-invoking).
- Top Performing Feature Sets: The "All Features" (combining Lexical, Grammatical, Document, and Theoretical), "Lexical," and "Theoretical" feature sets performed best, with statistically indistinguishable F1 scores.
- Informative Features: For negative cases (non-framing language), bigrams with quotation marks were informative. For positive cases (framing language), features related to proximity to punctuation, subjective terms, and bias terms were important.
Limitations and Future Work
- Precision-Recall Trade-off: The authors acknowledge the trade-off between precision and recall. While high recall is good for identifying potential framing, low precision might overwhelm users. Future work could explore different post-hoc decision thresholds to balance this.
- Context of Framing: The current study focuses on news articles, which may contain implicit or unconscious framing. Future work could compare these results with texts exhibiting more explicit framing (e.g., opinion columns, campaign speeches) and explore how framing identification differs across contexts.
- Expert vs. Lay Annotations: The study used lay annotators. Comparing these results with annotations from framing experts could provide further insights.
- Computational Cost: The authors note that full grammatical parsing, while informative, is computationally intensive. Future work could explore features that are quicker to extract for real-time applications.
- Role of Grammatical Structure: The surprisingly low performance of grammatical structure features warrants further investigation into their role in framing.
Summary updated