TL;DR
This paper investigates how users assess the credibility of tweets, particularly during fast-paced events like natural disasters, by examining the accuracy of their perceptions against expert evaluations. It reveals that while author details are crucial for accurate verification, the mere presence of corroborating tweets can paradoxically lead users to trust misinformation.
What Problem Does This Paper Address?
In the age of social media, Twitter has become a primary source for breaking news and real-time event updates. However, this rapid dissemination also facilitates the spread of misinformation and hoaxes, posing a significant challenge to users trying to discern truth from falsehood, especially during critical events where external verification is difficult. While previous research has explored what features users look at when assessing tweet credibility, this paper tackles the crucial question of how accurate those perceptions are and which features truly aid in verification.
Methods
The authors employ an epistemic approach, drawing on philosophical theories of knowledge and belief, to analyze tweet verification. They conducted a user study using Amazon Mechanical Turk (AMT) workers to assess credibility perceptions of tweets containing pictures related to Hurricane Sandy. AMT workers were tasked with rating various features of tweets (authority, text plausibility, picture plausibility, corroboration, and presentation) both individually and as a whole. These human-generated credibility ratings were then compared against ground truth labels determined by professional evaluations of the pictures' veracity.
Datasets
The study utilized a dataset of 332 tweets containing pictures related to Hurricane Sandy. These tweets were collected from Twitter's streaming API. The ground truth for the veracity of the pictures (real or fake) was established through professional assessments, which involved analyzing the pictures for authenticity, context, and potential manipulation.
Key Insights and Ideas
- Author Details Matter, But Are Underutilized: Access to more author information (like follower count, description, and location) significantly improves users' ability to accurately identify fake tweets. However, current Twitter interfaces don't prominently display these details.
- Corroboration Can Be Deceptive: Counterintuitively, seeing multiple tweets corroborating a piece of information, even if it's a hoax, can mislead users into believing it. This highlights the danger of relying solely on the quantity of supporting evidence without considering its independence.
- Presentation is a Poor Indicator: The writing quality, spelling, and grammar of a tweet (presentation) are not reliable indicators of its veracity, especially in the fast-paced, mobile-driven environment of Twitter.
- Emotional Resonance Trumps Truth: Users seem to share information that evokes an emotional response, regardless of its accuracy, mirroring findings in psychology about the spread of information that elicits strong feelings.
Key Results
- Authority as a Strong Predictor: When evaluating tweets solely based on author details, users achieved the highest precision in identifying fake tweets.
- Whole Tweet Evaluation is Best for Recall: Considering the entire tweet (author, text, and picture) provided the best recall for identifying real tweets, though it still misclassified a notable percentage of fake tweets as real.
- Picture Plausibility is Misleading: Fake pictures were often rated as highly plausible, especially when they depicted dramatic or unusual scenes, leading to misjudgments.
- Corroboration's Double-Edged Sword: While corroboration generally led to higher ratings for real tweets, it also significantly boosted ratings for fake tweets, with 71% of fake tweets receiving higher corroboration ratings than text plausibility ratings.
Limitations and Future Work
The authors acknowledge that their study focused on a specific event (Hurricane Sandy) and a particular type of content (tweets with pictures). They suggest future work should explore complementary approaches to digital content verification, including guidelines for journalists and computational methods that analyze implicit features like user proximity and content similarity within social graphs. They also propose developing reputation scores for users and enhancing features like independent corroboration and location identification to improve verification interfaces. Furthermore, they emphasize the need for user education on verification techniques, as simply knowing how to verify doesn't guarantee users will apply these methods.
Summary updated