Machine Learning Methods to Predict Social Media Disaster Rumor Refuters

Shihang Wang, Zongmin Li, Yuhong Wang, Qi ZhangInternational Journal of Environmental Research and Public Health 2019 · DOI

TL;DR

This paper introduces a machine learning framework to identify individuals likely to refute disaster-related rumors on social media. By combining traditional user features with natural language processing (NLP) analysis of recent microblog content, they significantly improve prediction accuracy, highlighting the importance of user sentiment and text similarity for identifying these crucial "anti-rumor spreaders."

What Problem Does This Paper Address?

The rapid spread of misinformation, especially during disasters, poses a significant threat to public safety and order. While many studies focus on tracking and predicting rumor propagation, fewer address how to actively combat it by identifying and leveraging individuals who actively debunk false information. This paper tackles the challenge of distinguishing "anti-rumor spreaders" from the general user base on social media platforms like Sina Weibo, aiming to develop a predictive model that can help proactively counter misinformation. The authors note that traditional methods often fail to convince users of the truth, and this research seeks to identify users who voluntarily spread refutations, potentially through their social connections.

Methods

The core of this research lies in developing a predictive model for identifying anti-rumor spreaders. They employed a multi-stage approach:

  • Data Collection: Microblogs related to four disaster events (earthquakes and fires) were collected from Sina Weibo.
  • Feature Engineering:
    • Traditional User Features: Basic information like gender, membership level, location, number of microblogs, followers, following, and group memberships were extracted.
  • NLP-based Features: This is a key innovation. They used Baidu's AipNLP to analyze the sentiment (positivity/negativity) and short text similarity of users' most recent microblog content. These analyses were converted into numerical values.
  • Machine Learning Models: Four classification algorithms were compared: Logistic Regression (LR), Support Vector Machines (SVM), Random Forest (RF), and Extreme Gradient Boosting (XGBoost). XGBoost emerged as the top performer.
  • Model Validation: The models were trained and tested on the collected data. They also investigated the generalization ability of the model across different disaster subclasses and the impact of group behavior and timeliness on prediction accuracy.

Datasets

The study utilized a dataset of 38,365 microblogs from 3,999 Sina Weibo users (3,793 valid users) related to four specific disaster events:

  1. Refutation of earthquake forecasting in Fujian, China.
  2. Refutation of a false video following an earthquake in Yibin, China.
  3. Refutation of a false explosion video in Queens, New York.
  4. Refutation of a conflagration at the Tencent Building in Beijing.

The data included microblog content, user profile information, and engagement metrics (retweets, comments, likes). The authors did not introduce a new benchmark dataset but rather curated and analyzed existing social media data.

Key Insights and Ideas

  • NLP is Crucial: The most significant finding is that incorporating NLP-derived features (sentiment and text similarity of recent posts) dramatically improves the prediction of anti-rumor spreaders compared to using only traditional user demographics. This suggests that what people are currently talking about and how they are talking about it are strong indicators of their willingness to engage with and spread refutations.
  • Recency Matters: The timeliness of microblog data is critical. Using the most recent posts (within the previous week) yielded significantly better and more stable prediction results than using posts from a month prior. This highlights that users' current interests and online behavior are more predictive than older activity.
  • Group Behavior is Less Influential than Expected: While group behavior variables (like the number of others who have retweeted, commented, or liked a post) were explored, they did not significantly improve prediction accuracy when added to the existing features. This suggests that individual-level content analysis is more powerful than simply observing crowd behavior for identifying spreaders.
  • Sina Weibo Specifics are Predictive: Features unique to platforms like Sina Weibo, such as verified information and user signatures/introductions, were found to be important predictors, underscoring the value of platform-specific contextual information.

Key Results

  • XGBoost Dominates: The XGBoost model consistently outperformed LR, SVM, and RF, achieving an average F1 Score of 0.747 with lower variance, indicating its robustness and effectiveness.
  • Improved Prediction: The inclusion of text analysis features led to a substantial increase in prediction robustness and precision. For instance, F1 scores for the "new" model (with text analysis) were generally higher and more stable across microblogs compared to the "traditional" model.
  • Generalization: The model showed good generalization ability within the same disaster subclass (e.g., earthquake to earthquake) and for the broader "major disaster" class. However, cross-subclass prediction (e.g., earthquake to fire/explosion) showed a decrease in performance, suggesting that while the model is adaptable within a domain, domain-specific features might be necessary for optimal performance across very different event types.
  • Feature Importance: Key predictors included sentiment and similarity of recent microblogs, the number of microblogs posted, verified information, signatures, and brief introductions. Less influential features were gender, location, and membership level.

Limitations and Future Work

  • Limited Disaster Events: The study was limited to four disaster events. The authors suggest examining more diverse disaster types and a larger sample size to further validate the model's generality.
  • Platform Specificity: While the model showed promise, its applicability to other platforms like WeChat and Facebook, which have different communication dynamics (e.g., closed-loop channels), needs further investigation. The authors propose optimizing and validating the methodology on such platforms.
  • Rumor Types: The current study focused on disaster-related rumors. Future work could explore the model's effectiveness with different types of rumors.
  • Timeliness Nuance: While "most recent" was crucial, the study substituted "one month after" for "one month before" due to data collection challenges. This substitution might not perfectly capture the immediate context, and future research could refine this aspect.
  • Group Behavior Nuance: Although group behavior variables didn't directly boost F1 scores, the authors noted their correlation with individual behavior, suggesting potential for more nuanced exploration in future work.

Summary updated

Abstract

This research provides a general methodology for distinguishing disaster-related anti-rumor spreaders from a non-ignorant population base, with strong connections in their social circle. Several important influencing factors are examined and illustrated. User information from the most recent posted microblog content of 3793 Sina Weibo users was collected. Natural language processing (NLP) was used for the sentiment and short text similarity analyses, and four machine learning techniques, i.e., logistic regression (LR), support vector machines (SVM), random forest (RF), and extreme gradient boosting (XGBoost) were compared on different rumor refuting microblogs; after which a valid and robust distinguishing XGBoost model was trained and validated to predict who would retweet disaster-related rumor refuting microblogs. Compared with traditional prediction variables that only access user information, the similarity and sentiment analyses of the most recent user microblog contents were found to significantly improve prediction precision and robustness. The number of user microblogs also proved to be a valuable reference for all samples during the prediction process. This prediction methodology could be possibly more useful for WeChat or Facebook as these have relatively stable closed-loop communication channels, which means that rumors are more likely to be refuted by acquaintances. Therefore, the methodology is going to be further optimized and validated on WeChat-like channels in the future. The novel rumor refuting approach presented in this research harnessed NLP for the user microblog content analysis and then used the analysis results of NLP as additional prediction variables to identify the anti-rumor spreaders. Therefore, compared to previous studies, this study presents a new and effective decision support for rumor countermeasures.
BibTeX
@article{wang2019machine,
  author = {Shihang Wang and Zongmin Li and Yuhong Wang and Qi Zhang},
  title = {{Machine Learning Methods to Predict Social Media Disaster Rumor Refuters}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2019},
  doi = {10.3390/ijerph16081452},
}
Figure