ACCD: An Adaptive Clustering-Based Collusion Detector in Crowdsourcing (Student Abstract)

Ruoyu Xu, Gaoxiang Li, Wei Jin, Austin Chen, Victor S. ShengAAAI 2023 · DOI

Introduction

Crowdsourcing platforms leverage the "wisdom of the crowd" to solve complex tasks, ranging from image labeling to subjective product evaluations. However, this collective intelligence is increasingly threatened by collusion—coordinated efforts where participants synchronize their answers to maximize rewards with minimal effort. Such behavior not only wastes resources but also poisons datasets with biased or incorrect information.

Detecting collusion is challenging due to two primary factors. First, existing tools are often rigid, designed primarily for "ground truth" tasks with objective answers and failing in subjective contexts like sentiment analysis. Second, many systems rely on manual thresholds, requiring administrators to pre-define suspicious behavior patterns, which is impractical as collusion tactics evolve. To address these issues, the Adaptive Clustering-Based Collusion Detector (ACCD) introduces a flexible, density-based approach that identifies colluding groups across diverse task types without requiring prior parameter tuning.

Methodology: Leveraging Density-Based Clustering

The core of ACCD is built upon HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise). The fundamental intuition is that while honest workers' responses follow a natural, often noisy distribution, colluders create unnaturally dense clusters in the answer space.

The HDBSCAN-based clustering process used in ACCD to identify collusion groups.

The algorithm utilizes two critical metrics to ensure robustness:

  • Core Distance: Defined as the distance between a data point and its $k^{th}$ nearest neighbor, serving as a measure of local density.
  • Mutual Reachable Distance: A metric that effectively separates noise from signal by pushing points in sparse regions further apart while maintaining the proximity of points in dense regions.

By constructing a hierarchy of clusters based on these distances, ACCD can adapt to varying data distributions. Unlike traditional K-means clustering, it does not require a pre-specified number of clusters, allowing it to discover collusion groups of arbitrary sizes and densities. This hierarchical approach enables the system to distinguish between the "tight" clusters of colluders and the more dispersed "noise" of honest participants.

Experimental Setup and Datasets

The researchers evaluated ACCD's performance using a combination of real-world and extensive synthetic datasets:

  • Real-World Data: The study utilized an e-commerce product rating dataset comprising 20 tasks and 123 participants. Notably, this dataset includes 36 confirmed colluders who admitted to coordinating their responses, providing a rare "ground truth" for evaluating detection accuracy in subjective tasks.
  • Synthetic Data: To rigorously test the algorithm's versatility, the authors generated 4,000 simulated datasets. These covered four distinct scenarios: categorical ratings (subjective), continuous ratings (subjective), categorical ground truth (objective), and continuous ground truth (objective).

Key Insights and Results

The evaluation revealed several key advantages of the ACCD approach. By treating collusion as a density problem rather than a simple similarity problem, the algorithm successfully identifies "spikes" of high density that characterize coordinated cheating.

Performance Comparison

In head-to-head comparisons with state-of-the-art methods such as FINDCOLLUDERS (FC), Collusion-Proof (CP), and PROCAP, ACCD demonstrated superior performance:

  • Accuracy and F1 Score: On real-world data, ACCD achieved an F1 Score of 0.883 and an Accuracy of 0.927, significantly outperforming the next best method, FINDCOLLUDERS, which reached an F1 Score of 0.816.
  • High Recall: In synthetic testing, ACCD maintained a recall rate between 0.93 and 1.00, indicating its exceptional ability to capture nearly all colluders present in a system.
  • Cross-Task Versatility: While traditional methods often saw performance drops when moving from objective to subjective tasks, ACCD maintained high accuracy (above 0.95) across all four tested problem types.

Discussion and Future Directions

ACCD represents a significant step toward automated, adaptive quality control in crowdsourcing. Its ability to function without manual thresholds makes it highly practical for real-world deployment where collusion patterns are unpredictable.

However, as a student abstract, several avenues for future research remain. While HDBSCAN is computationally efficient, further study is needed to assess how ACCD scales to massive platforms with millions of submissions. Additionally, the current model assumes colluders provide highly similar answers; future iterations could investigate "sophisticated" colluders who introduce artificial noise to evade density-based detection. Finally, exploring the sensitivity of the min_cluster_size parameter in diverse real-world scenarios would further refine the system's adaptivity.

Summary updated

BibTeX
@inproceedings{xu2023accd,
  author = {Ruoyu Xu and Gaoxiang Li and Wei Jin and Austin Chen and Victor S. Sheng},
  title = {{ACCD: An Adaptive Clustering-Based Collusion Detector in Crowdsourcing (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27045},
}
Figure