TDG4Crowd: Test Data Generation for Evaluation of Aggregation Algorithms in Crowdsourcing

Yili Fang, Chaojie Shen, Huamao Gu, Tao Han, Xinyi DingIJCAI 2023 · DOI

TL;DR

This paper introduces TDG4Crowd, a novel method for automatically generating comprehensive and balanced synthetic datasets for evaluating crowdsourcing aggregation algorithms. By learning latent representations of annotators and instances, TDG4Crowd addresses the inconsistencies observed when evaluating these algorithms on sparse, real-world crowdsourced data, leading to more reliable algorithm comparisons.

What Problem Does This Paper Address?

Crowdsourcing platforms like Amazon Mechanical Turk are widely used for tasks requiring human intelligence, such as image annotation. To derive a single "ground truth" from multiple noisy labels provided by different workers, aggregation algorithms are employed. However, evaluating these algorithms is challenging because real-world crowdsourced datasets are often sparse and imbalanced due to budget constraints. This sparsity can lead to contradictory conclusions about algorithm performance. For instance, an algorithm that excels on one dataset might perform poorly on another. Existing research often relies on these problematic real datasets or theoretical analyses that may not fully capture real-world complexities. The paper aims to bridge this gap by creating a method to generate synthetic datasets that are both comprehensive and balanced, allowing for fairer and more consistent evaluation of aggregation algorithms.

Methods

TDG4Crowd is a generative approach that learns latent representations for annotators and instances. It comprises three main modules:

  1. Annotator Model: This module uses a Variational Autoencoder (VAE) to learn a latent variable ($z_a$) for each annotator. The VAE learns a distribution that approximates the annotator's characteristics, aiming to capture their skill and potential biases. The loss function includes a reconstruction loss and a KL divergence term to regularize the latent space.
  2. Instance Model: Similar to the annotator model, this module also employs a VAE to learn a latent variable ($z_i$) for each instance. For image data, pre-trained VGG-16 features are used as input. The goal is to capture the inherent difficulty or characteristics of each instance.
  3. Inferring Component: This component takes the learned latent representations of an annotator ($z_a$) and an instance ($z_i$) and predicts the annotation ($\hat{y}_n$) for that instance. This is done using a fully connected network trained in a supervised manner, aiming to reconstruct the original annotation.

The overall training process involves first training the annotator and instance models separately, then integrating the inferring component to train the entire system end-to-end.

Datasets

The paper uses two real-world datasets for evaluation:

  • LabelMe: A dataset for image annotation.
  • Relation: Another dataset for image annotation.

The authors also mention using annotations collected through the Appen and AMT crowdsourcing platforms. No new benchmark datasets are introduced; instead, the focus is on generating synthetic data that mimics the distribution of these real datasets.

Key Insights and Ideas

  • Dataset Imbalance is a Major Hurdle: The core insight is that the sparsity and imbalance inherent in real crowdsourced datasets significantly hinder fair and consistent evaluation of aggregation algorithms, leading to conflicting results.
  • Latent Representation is Key: Learning rich latent representations for both annotators and instances is crucial for generating realistic and diverse synthetic data.
  • VAE for Generative Power: The use of Variational Autoencoders is a powerful technique for learning these latent distributions and enabling the generation of new data points.
  • Consistency Matters: Beyond just matching data distributions (KL/KS divergence), the paper emphasizes that the consistency of aggregation algorithm rankings across different datasets is a critical evaluation metric, which TDG4Crowd aims to improve.

Key Results

  • Superior Data Distribution Matching: TDG4Crowd consistently achieved lower KL divergence and KS test values compared to baseline methods (G_DS, G_HDS, G_MV, G_GLAD, G_IRT) across various scenarios (generating instances, annotators, or both). This indicates that the synthetic data generated by TDG4Crowd more closely matches the distribution of real datasets.
  • Improved Aggregation Algorithm Consistency: Experiments showed that aggregation algorithms performed more consistently on the synthetic datasets generated by TDG4Crowd. Specifically, the rankings of algorithms (e.g., DS, HDS, MV, GLAD) on synthetic data were more aligned with their rankings on real data, addressing the core motivation of the paper.
  • Robustness Across Scenarios: TDG4Crowd demonstrated strong performance across different data generation scenarios, including generating new instances, new annotators, or both, even when dealing with limited real data.

Limitations and Future Work

  • Subjective Judgment for Consistency: The paper notes that direct metrics for evaluating the consistency of aggregation algorithm rankings are lacking, and subjective judgment was used in the experiments.
  • Simple Annotation Tasks: The current work focuses on simple annotation tasks.
  • Future Work: The authors suggest investigating more challenging tasks that involve context information for future work. They also aim to further explore the consistency analysis of aggregation algorithms.

Summary updated

BibTeX
@inproceedings{fang2023tdg,
  author = {Yili Fang and Chaojie Shen and Huamao Gu and Tao Han and Xinyi Ding},
  title = {{TDG4Crowd: Test Data Generation for Evaluation of Aggregation Algorithms in Crowdsourcing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/333},
}
Figure