TL;DR
This paper introduces "Composite Crowd Tasks" (CCTs), a novel framework for designing and aggregating labels from multiple, potentially heterogeneous, crowd tasks. They propose a unified label aggregation method that leverages shared worker information across these tasks to improve overall label quality, outperforming existing methods designed for single, homogeneous tasks.
What Problem Does This Paper Address?
Crowdsourcing is a powerful tool for data labeling, but existing methods primarily focus on "Single Crowd Tasks" (SCTs) where workers perform a single type of task (e.g., classification, pairwise comparison) on a single set of objects. In real-world scenarios, requesters often need to combine multiple, diverse tasks (e.g., categorization and ranking) or tasks on different object sets into a single "Composite Crowd Task" (CCT). The challenge is how to effectively aggregate labels from these heterogeneous CCTs, especially when workers might contribute to multiple tasks within the CCT. Simply aggregating each SCT independently ignores valuable cross-task information, particularly regarding worker abilities. This paper addresses this gap by proposing a method to bridge these diverse SCTs within a CCT, leveraging shared worker information to enhance label quality.
Methods
The core of the paper is a novel label aggregation approach for CCTs that unifies existing SCT models under a multi-task objective function. Key aspects include:
- Backbone Model Reformulation: Existing label aggregation models for SCTs (like GLAD for categorization, CROWDBT/HBTL for pairwise preference, and Gaussian/Student-t kernels for similarity) are reformulated into a unified format. This allows them to be seamlessly integrated into the CCT framework.
- Worker Ability Constraint Satisfaction: The central idea is to leverage the fact that workers might contribute to multiple SCTs within a CCT. The authors propose a worker ability constraint satisfaction mechanism. If a worker participates in two SCTs, their estimated abilities are constrained to be similar across these tasks. This constraint can be relaxed using various distance metrics (Euclidean, KL-divergence, L1 norm) or even rank-based constraints.
- Multi-Task Objective Function: The reformulated SCT models are combined into a single objective function for the CCT. This function includes the individual SCT loss terms and a regularization term that enforces the worker ability constraints. This allows for joint optimization of all parameters, including worker abilities and true labels, across all SCTs in the CCT.
Datasets
The authors created novel real crowd datasets for CCTs to evaluate their approach, as existing datasets were insufficient. These datasets cover three main cases:
- Case ①: CCTs with pairwise preference and pairwise similarity tasks for the same object set. Two datasets were created based on "cheat" and "meeting" opinion datasets.
- Case ②: CCTs with two preference tasks or two similarity tasks on different object sets. Datasets were created using the "cheat" and "meeting" opinion sets.
- Case ③: CCTs with a categorization task and a pairwise preference task on different object sets. This case involves tasks like identifying clusters in point sets and comparing cluster visualizations.
The paper provides links to these datasets and associated code.
Key Insights and Ideas
- Bridging Heterogeneity: The most significant insight is that worker abilities are a crucial bridge across diverse crowd tasks within a CCT. By modeling and constraining these abilities, information from one task can inform and improve another.
- Unified Framework: Reformulating existing SCT models into a common format is key to enabling their integration into a single CCT model. This modular approach allows for flexibility in choosing backbone models.
- Worker Ability as a Global Signal: Instead of treating worker abilities as task-specific, the CCT framework treats them as a global signal that can be shared and refined across multiple tasks, leading to more robust estimates.
- Relaxed Constraints for Practicality: Recognizing that worker abilities might not be perfectly identical across tasks, the use of relaxed optimization and various constraint types (distance-based, rank-based) makes the approach more practical.
Key Results
The proposed CCT label aggregation method consistently outperformed baseline SCT models across various tasks and datasets.
- Preference and Similarity Tasks (Cases ① & ②): The method showed statistically significant improvements in pairwise preference accuracy, outperforming baselines like HBTL and CROWDBT. The different constraint types (EUC, KL, L1, Rank) generally yielded comparable results, with EUC and L1 showing slight advantages in achieving the best results.
- Categorization and Preference Tasks (Case ③): For the categorization task, the proposed method achieved perfect accuracy, indicating the ease of estimating labels and worker abilities in such scenarios. Crucially, it significantly improved the pairwise preference accuracy in the second task, demonstrating the benefit of bridging tasks.
Limitations and Future Work
- Assumptions on Worker Abilities: The approach assumes that workers in different SCTs within a CCT have similar abilities, especially when they contribute to multiple tasks. While relaxed constraints are offered, this assumption might not hold in all scenarios.
- CCT Design Policies: The paper mentions that CCT design policies ensure comparable difficulties and worker abilities. Future work could explore how to handle CCTs where these assumptions are less strictly met.
- Complexity of CCTs: The current work focuses on CCTs with two SCTs. The authors suggest that the framework can be generalized to more than two tasks, and future work could explore more complex CCT designs and scenarios.
- Other Domains: The current experiments cover specific types of tasks. Future work could extend the approach to other domains and more complex task types.
Summary updated