TL;DR
During crises, people struggle to find reliable information amidst social media noise. This paper introduces an AI-driven "intent classification model" that helps users triage and understand crisis-related information by categorizing tweets based on their communicative purpose, bridging the gap between what users seek and what they truly need.
What Problem Does This Paper Address?
In times of crisis, individuals are often overwhelmed with information, especially from social media. While platforms like Twitter can be valuable for disseminating information from experts, it's incredibly difficult for lay users to distinguish credible, actionable advice from the sheer volume of opinions, rumors, and irrelevant content. This "information foraging" can lead to confusion, anxiety, and potentially counterproductive actions, as seen during the COVID-19 pandemic. Existing approaches to fact-checking are often external and not user-controlled, and users may not even know what information they need to seek in an unfolding, uncertain situation. This paper aims to empower users by providing them with tools to better navigate and make sense of this complex information landscape.
Methods
The core of this work is an intent classification model (ICM) designed to categorize social media messages based on their communicative purpose during a crisis. This model is built upon insights from existing crisis communication theories, including the Health Belief Model (HBM), the Social Mediated Crisis Communication Model (SMCC), and the Elaboration Likelihood Model (ELM).
The methodology involves several key steps:
- Topic Modeling: Using a BERT-based model (BERTopic), they identified 25 topic categories from COVID-19 related tweets, such as "vaccine development," "scientific information," and "mitigation."
- Subjectivity and Sentiment Analysis: They trained a DistilBERT model to classify tweets as objective, subjective, or irrelevant, and used a pre-trained BERT-based sentiment analysis model to determine positive, negative, or neutral sentiment.
- Intent Classification: This is the novel contribution. They manually labeled 6,844 tweets into five intent categories:
- Expert Guidance: Suggestions or recommendations from experts.
- Situational Awareness: Updates on news, policies, or general proceedings.
- Severity: Reports on the impact or scale of the crisis.
- Reactions: Emotions, comments, or responses to crisis events.
- Lived Experience: Personal accounts and direct experiences.
A DistilBERT model was trained on these labeled tweets, achieving an overall accuracy of approximately 70%.
Datasets
The primary dataset used is a collection of tweets from March 2020 to September 2020, focusing on COVID-19. This dataset was curated in two batches:
- Expert Tweets: Collected from a list of 50 health and science experts identified by Elemental, a health and wellness publication. These were verified for credibility by a medical sciences domain researcher.
- General User Tweets: Collected from general social media users, with profiles carefully filtered to exclude those who might be considered experts (e.g., "virologist," "clinician").
For topic modeling, a subset of 5,000 expert tweets was used. For subjectivity analysis, 10,000 tweets were labeled. For intent classification, 6,844 tweets were labeled.
Key Insights and Ideas
- Experts are not always purely objective: Contrary to expectations, experts also exhibited subjectivity in their tweets, highlighting the nuanced nature of communication even from authoritative sources.
- User agency is crucial: The paper emphasizes the need to give social media users more control over their information feeds, moving beyond passive consumption to active curation and sensemaking.
- Intent is a powerful lens: Categorizing information by its communicative intent (e.g., guidance vs. personal experience) provides a more effective way for users to triage and understand information than just topic or sentiment alone.
- Lived experience can be highly valuable: The hypothetical user scenario demonstrates how personal accounts from lay users can offer unexpected but crucial insights (e.g., the viability of online learning) that might be missed by focusing solely on expert pronouncements.
Key Results
- The intent classification model achieved an overall accuracy of approximately 70%, with specific intent categories showing higher accuracy (e.g., Severity at 93%, Lived Experience at 80%).
- The model's performance on real-world data suggests its potential for learning from user interactions and adapting to evolving information needs.
- A hypothetical user workflow (Figure 3) illustrates how the intent classification model, integrated into a web interface, can help a parent navigate complex information about homeschooling during the pandemic, enabling them to make informed decisions by filtering and summarizing relevant content.
Limitations and Future Work
- Limitations: The authors acknowledge that current content moderation methods don't allow users to update the underlying models. The accuracy of the intent classification model, while promising, could still be improved.
- Future Work: The authors plan to develop a software prototype of the human-AI interface to allow lay users to explore messages by author intent and control their information exposure. They also intend to conduct user studies with diverse participants to evaluate the usefulness and utility of their proposed sensemaking workflow and interface. Further development will focus on incorporating end-user content moderation techniques.
Summary updated