Abstract
Twitter has emerged as a new application paradigm of sensing the physical environment by using human as sensors. These human sensed observations are often viewed as binary claims (either true or false). A fundamental challenge on Twitter is how to ascertain the credibility of claims and the reliability of sources without the prior knowledge on either of them beforehand. This challenge is referred to as truth discovery. An important limitation exists in the current Twitter-based truth discovery solutions: they did not explore the theme relevance aspect of claims and the correct claims identified by their solutions can be completely irrelevant to the theme of interests. In this paper, we present a new analytical model that explicitly considers the theme relevance feature of claims in the solutions of truth discovery problem on Twitter. The new model solves a bi-dimensional estimation problem to jointly estimate the correctness and theme relevance of claims as well as the reliability and theme awareness of sources. The new model is compared with the discovery solutions in current literature using three real world datasets collected from Twitter during recent disastrous and emergent events: Paris attack, Oregon shooting, and Baltimore riots, all in 2015. The new model was shown to be effective in terms of finding both correct and relevant claims.
Introduction: The Challenge of Social Sensing
In the wake of a disaster—like a mass shooting or a terrorist attack—Twitter becomes a vital "social sensor" network. However, this sensor network is incredibly noisy. Truth discovery is the process of determining which claims are credible without knowing beforehand which sources are reliable.
Existing truth-discovery algorithms focus almost exclusively on whether a claim is true or false, ignoring whether it is relevant. During a crisis, people often use trending hashtags to post jokes, advertisements, or personal updates. A traditional model might correctly identify a tweet as "true" (e.g., "I am eating a sandwich #ParisAttacks"), but that information is useless to emergency responders. This research seeks to isolate "valuable" claims—those that are both factually correct and strictly relevant to the event at hand.
Methodology: The TR-EM Scheme
The proposed Theme-Relevant Expectation Maximization (TR-EM) scheme is an estimation-theoretic approach that treats the problem as a bi-dimensional challenge.
Key components include:
- Bi-dimensional Latent Variables: The model simultaneously estimates two hidden properties for every claim: its correctness (True/False) and its theme relevance (Relevant/Irrelevant).
- Source Modeling: Instead of a single "reliability" score, the model calculates two distinct metrics for each Twitter user: Reliability (likelihood of reporting correct claims) and Theme Awareness (likelihood of staying on-topic).
- The EM Algorithm: Using an iterative Expectation-Maximization (EM) process, the model starts with an initial guess, observes user agreement, and iteratively refines its estimates of claim-truth and source-reliability until convergence.
Experimental Evaluation
The researchers evaluated TR-EM using three high-profile, real-world datasets from 2015:
- Paris Terrorist Attacks: ~873,000 tweets from ~496,000 users.
- Oregon Umpqua Community College Shooting: ~210,000 tweets from ~122,000 users.
- Baltimore Riots: ~952,000 tweets from ~425,000 users.
These datasets are particularly challenging because they contain massive amounts of conflicting, incomplete, and noisy data generated in high-pressure scenarios.
Key Insights and Results
A critical insight of this work is that hashtags are insufficient for relevance. Irrelevant tweets often use trending hashtags for visibility, while relevant ones might omit them. TR-EM learns to identify relevance by looking at source behavior rather than just keywords. By defining a "valuable claim" as the intersection of truth and relevance, the authors provide a more practical tool for disaster response.
TR-EM was compared against several baselines, including simple voting, hashtag filtering, and "Regular EM" (which ignores relevance).
- Superior Performance: In the Paris Attack dataset, TR-EM achieved an 8% gain in precision and a 20% gain in F1-measure over the best-performing baselines. In the Oregon shooting data, the model showed a 23% improvement in F1-measure.
- Efficiency: The algorithm typically converges within just a few iterations, making it suitable for near-real-time applications.
Conclusion and Future Work
While TR-EM significantly improves truth discovery, it currently treats claims as binary (True/False). Future work could explore multi-valued facts or more complex features for relevance. Nevertheless, the joint estimation of reliability and theme awareness marks a significant step forward in extracting valuable signal from the noise of social media.
Summary updated