Performance as a Constraint: An Improved Wisdom of Crowds Using Performance Regularization

Jiyi Li, Yasushi Kawase, Yukino Baba, Hisashi Kashima

Introduction

Crowdsourcing often relies on the "Wisdom of Crowds"—the principle that aggregating multiple independent judgments reveals the truth. However, this principle falters on tasks requiring specialized expertise, such as medical diagnosis or technical translation. In these scenarios, non-experts may rely on the availability heuristic, leading the majority to converge on a familiar but incorrect answer. Traditional quality control methods, whether unsupervised (e.g., Dawid-Skene) or supervised (using qualification tests), typically weight worker contributions "locally" on a per-question basis. This paper argues that such approaches overlook a critical signal: the global consistency of a worker's performance across an entire task set.

Methodology: Performance as a Global Constraint

The authors introduce a novel framework that treats worker performance not merely as a weight, but as a global constraint on the inferred solution. This approach, termed Performance Constraint (PC), frames truth inference as a constraint satisfaction problem: if a worker's estimated accuracy is 70%, the final set of inferred "true" answers must be consistent with that worker being correct exactly 70% of the time across all tasks.

To handle real-world uncertainty in performance estimation, the authors propose several relaxations:

  • Distance-based (PC-EUC/PC-KL): Minimizes the Euclidean distance or KL-divergence between estimated and observed performance.
  • Ranking-based (PC-RANK): A robust alternative that assumes only the relative order of worker expertise (e.g., if Worker A outperformed Worker B on a test, they should also do so on the main task).

Furthermore, the authors extend this concept into Performance Regularization (PR), integrating these constraints as regularizers into existing state-of-the-art probabilistic models like GLAD or MINMAX. This allows models to benefit from both local probabilistic inference and global performance consistency.

Experimental Setup

The proposed methods were evaluated on six diverse real-world datasets featuring "heterogeneous multiple-choice questions" (where the number of options varies):

  • Language: Chinese and English.
  • Technical/Specialized: Information Technology (IT), Medicine, and Science.
  • Niche Knowledge: Pokémon.

These datasets were specifically selected for their difficulty, as evidenced by the poor performance of simple Majority Voting (MV).

Key Results

Superior Standalone Performance

When worker performance is known accurately, the PC method demonstrates remarkable efficacy. As shown in the table below, PC variants achieved perfect or near-perfect accuracy across all datasets, significantly outperforming traditional unsupervised methods like Dawid-Skene (D&S) and GLAD.

Accuracy comparison of PC variants against baseline methods across six datasets.

Noise Tolerance and Robustness

A critical challenge in this framework is the potential for "noisy" performance estimates. The authors tested the robustness of their methods by introducing bias and standard deviation into the performance estimates. The ranking-based relaxation (PC-RANK) proved particularly resilient, maintaining high accuracy even when the exact performance values were inaccurate, provided the relative ranking of workers remained stable.

Robustness of PC methods to noise in worker performance estimation.

Boosting Existing Models

The Performance Regularization (+PR) extension consistently improved the performance of standard models. For instance, on the "Medicine" dataset, adding PR to the GLAD model increased its accuracy from 0.811 to 0.967, demonstrating the value of global constraints as a complementary signal to local probabilistic modeling.

Conclusion

By shifting the perspective of worker performance from a local weight to a global constraint, this work provides a powerful tool for improving truth inference in difficult crowdsourcing tasks. While the method's effectiveness depends on the quality of initial performance estimates (e.g., via qualification questions), the ranking-based approach offers significant robustness. Future work may explore applying these constraints to more complex generative models or scenarios with sparse worker performance data.

Summary updated

Abstract

Quality assurance is one of the most important problems in crowdsourcing and human computation, and it has been extensively studied from various aspects. Typical approaches for quality assurance include unsupervised approaches such as introducing task redundancy (i.e., asking the same question to multiple workers and aggregating their answers) and supervised approaches such as using worker performance on past tasks or injecting qualification questions into tasks in order to estimate the worker performance. In this paper, we propose to utilize the worker performance as a global constraint for inferring the true answers. The existing semi-supervised approaches do not consider such use of qualification questions. We also propose to utilize the constraint as a regularizer combined with existing statistical aggregation methods. The experiments using heterogeneous multiple-choice questions demonstrate that the performance constraint not only has the power to estimate the ground truths when used by itself, but also boosts the existing aggregation methods when used as a regularizer.
BibTeX
@inproceedings{li2020performance,
  author = {Jiyi Li and Yasushi Kawase and Yukino Baba and Hisashi Kashima},
  title = {{Performance as a Constraint: An Improved Wisdom of Crowds Using Performance Regularization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/213},
}
Figure