Introduction
Crowdsourcing often relies on the "Wisdom of Crowds"—the principle that aggregating multiple independent judgments reveals the truth. However, this principle falters on tasks requiring specialized expertise, such as medical diagnosis or technical translation. In these scenarios, non-experts may rely on the availability heuristic, leading the majority to converge on a familiar but incorrect answer. Traditional quality control methods, whether unsupervised (e.g., Dawid-Skene) or supervised (using qualification tests), typically weight worker contributions "locally" on a per-question basis. This paper argues that such approaches overlook a critical signal: the global consistency of a worker's performance across an entire task set.
Methodology: Performance as a Global Constraint
The authors introduce a novel framework that treats worker performance not merely as a weight, but as a global constraint on the inferred solution. This approach, termed Performance Constraint (PC), frames truth inference as a constraint satisfaction problem: if a worker's estimated accuracy is 70%, the final set of inferred "true" answers must be consistent with that worker being correct exactly 70% of the time across all tasks.
To handle real-world uncertainty in performance estimation, the authors propose several relaxations:
- Distance-based (PC-EUC/PC-KL): Minimizes the Euclidean distance or KL-divergence between estimated and observed performance.
- Ranking-based (PC-RANK): A robust alternative that assumes only the relative order of worker expertise (e.g., if Worker A outperformed Worker B on a test, they should also do so on the main task).
Furthermore, the authors extend this concept into Performance Regularization (PR), integrating these constraints as regularizers into existing state-of-the-art probabilistic models like GLAD or MINMAX. This allows models to benefit from both local probabilistic inference and global performance consistency.
Experimental Setup
The proposed methods were evaluated on six diverse real-world datasets featuring "heterogeneous multiple-choice questions" (where the number of options varies):
- Language: Chinese and English.
- Technical/Specialized: Information Technology (IT), Medicine, and Science.
- Niche Knowledge: Pokémon.
These datasets were specifically selected for their difficulty, as evidenced by the poor performance of simple Majority Voting (MV).
Key Results
Superior Standalone Performance
When worker performance is known accurately, the PC method demonstrates remarkable efficacy. As shown in the table below, PC variants achieved perfect or near-perfect accuracy across all datasets, significantly outperforming traditional unsupervised methods like Dawid-Skene (D&S) and GLAD.
Noise Tolerance and Robustness
A critical challenge in this framework is the potential for "noisy" performance estimates. The authors tested the robustness of their methods by introducing bias and standard deviation into the performance estimates. The ranking-based relaxation (PC-RANK) proved particularly resilient, maintaining high accuracy even when the exact performance values were inaccurate, provided the relative ranking of workers remained stable.
Boosting Existing Models
The Performance Regularization (+PR) extension consistently improved the performance of standard models. For instance, on the "Medicine" dataset, adding PR to the GLAD model increased its accuracy from 0.811 to 0.967, demonstrating the value of global constraints as a complementary signal to local probabilistic modeling.
Conclusion
By shifting the perspective of worker performance from a local weight to a global constraint, this work provides a powerful tool for improving truth inference in difficult crowdsourcing tasks. While the method's effectiveness depends on the quality of initial performance estimates (e.g., via qualification questions), the ranking-based approach offers significant robustness. Future work may explore applying these constraints to more complex generative models or scenarios with sparse worker performance data.
Summary updated