The Dark Side of Micro-Task Marketplaces: Characterizing Fiverr and Automatically Detecting Crowdturfing

Kyumin Lee, Steve Webb, Hancheng Ge

TL;DR

This paper investigates "crowdturfing"—the buying and selling of fake online engagement—on the micro-task platform Fiverr. The authors characterize the market, build a highly accurate machine learning classifier to detect malicious gigs, and demonstrate the real-world impact of these services by purchasing fake Twitter followers that successfully evaded Twitter's security systems.

What Problem Does This Paper Address?

Crowdsourcing platforms like Amazon Mechanical Turk and Fiverr are incredibly useful for distributing micro-tasks to a global workforce. However, they have a dark side: malicious actors use these platforms to buy and sell fake reviews, artificial social media followers, and manipulated search engine backlinks. This practice is known as "crowdturfing" (crowdsourced astroturfing).

While previous research noted the existence of crowdturfing, little was known about the specific dynamics of the Fiverr marketplace, the demographics of its users, or how to automatically detect and filter out these malicious tasks. The authors tackle this gap by analyzing the Fiverr ecosystem and building a system to automatically detect crowdturfing gigs before they can inflict damage on the broader web ecosystem.

Methods

The researchers built a custom crawler to scrape Fiverr, collecting gig listings and user profiles. They manually labeled a subset of 1,550 gigs as either legitimate or crowdturfing.

Using this labeled data, they extracted features to train a classifier. These features included gig titles and descriptions (using TF-IDF bag-of-words), categories, seller ratings, and seller demographics (like country and longevity). They tested over 30 machine learning algorithms—including Naive Bayes, Support Vector Machines (SVM), and tree-based models—to classify the gigs.

Finally, they conducted a real-world impact study. They purchased five Twitter follower gigs for newly created, empty Twitter accounts and tracked the resulting follower delivery speed, retention, and influence metrics (Klout scores) to see if target platforms could actually detect the manipulation.

Datasets

  • Fiverr Crawl Dataset: A newly collected dataset of 89,667 active gig listings and 31,021 user profiles scraped between July and August 2013.
  • Labeled Subset: 1,550 randomly selected gigs manually labeled for training and evaluation (121 crowdturfing, 1,429 legitimate).

Key Insights and Ideas

  • Crowdturfing is highly lucrative: The top Fiverr seller in the dataset earned an estimated $3 million over two years selling SEO manipulation (backlink) gigs. This highlights a massive financial incentive driving web manipulation.
  • Western dominance in the click-farm economy: Unlike other micro-task sites where workers are primarily from developing nations, the majority of Fiverr crowdturfing sellers (35.8%) and buyers are from the United States. Fiverr's $5 minimum price point makes it an attractive hustle for Western users compared to platforms that pay pennies per task.
  • Three main flavors of manipulation: Crowdturfing gigs primarily target social media (70.7% - e.g., fake Facebook likes/Twitter followers), search engines (27.3% - e.g., artificial backlinks to boost PageRank), and direct user traffic (2% - e.g., bot traffic to farm ad clicks).
  • Target platforms are failing at detection: The authors proved that buying influence is shockingly easy and effective. Furthermore, the targeted platforms' native security systems are inadequate at catching these artificially generated crowds.

Key Results

  • Detection Accuracy: The SVM-based classifier performed the best, achieving an impressive 97.35% accuracy with an F1 score of 0.974 and a False Positive Rate of just 0.008.
  • Market Scale: Applying the classifier to the full dataset revealed that 19,904 gigs—22.2% of the active marketplace—were crowdturfing tasks.
  • Real-World Impact: In the live experiment, purchasing just $25 worth of gigs yielded 46,176 Twitter followers within 48 hours. This artificially inflated the target accounts' Klout (influence) scores from 0 to nearly 20.
  • Security Failure: Two months after the fake followers were delivered, Twitter's security team had only suspended 24.6% of them, leaving the vast majority of the purchased influence intact.

Limitations and Future Work

The authors note that their data collection was slightly limited by Fiverr's anti-scraping mechanisms (IP blocks), requiring them to use human-readable URLs instead of sequential IDs, though they still captured an estimated 84% of active gigs.

While the offline classifier proved highly accurate, the authors suggest that future work should focus on widely deploying such detection systems in real-time. By integrating these classifiers directly into micro-task marketplaces or the security backends of targeted sites, platforms could proactively prevent these tasks from being purchased and executed.

Summary updated

Abstract

As human computation on crowdsourcing systems has become popular and powerful for performing tasks, malicious users have started misusing these systems by posting malicious tasks, propagating manipulated contents, and targeting popular web services such as online social networks and search engines. Recently, these malicious users moved to Fiverr, a fast-growing micro-task marketplace, where workers can post crowdturfing tasks (i.e., astroturfing campaigns run by crowd workers) and malicious customers can purchase those tasks for only $5. In this paper, we present a comprehensive analysis of Fiverr. First, we identify the most popular types of crowdturfing tasks found in this marketplace and conduct case studies for these crowdturfing tasks. Then, we build crowdturfing task detection classifiers to filter these tasks and prevent them from becoming active in the marketplace. Our experimental results show that the proposed classification approach effectively detects crowdturfing tasks, achieving 97.35% accuracy. Finally, we analyze the real world impact of crowdturfing tasks by purchasing active Fiverr tasks and quantifying their impact on a target site. As part of this analysis, we show that current security systems inadequately detect crowdsourced manipulation, which confirms the necessity of our proposed crowdturfing task detection approach.
BibTeX
@inproceedings{lee2014dark,
  author = {Kyumin Lee and Steve Webb and Hancheng Ge},
  title = {{The Dark Side of Micro-Task Marketplaces: Characterizing Fiverr and Automatically Detecting Crowdturfing}},
  booktitle = {ICWSM},
  year = {2014},
  url = {https://openreview.net/forum?id=Z3j34YYC8w},
}

Loading related content...

Figure