Abstract
Crowdsourcing provides a cost-effective solution for labeling large-scale datasets, yet it is inherently prone to human error and inconsistency. Traditional truth inference methods often overlook the relationship between item features and worker performance. This paper introduces MMLC (Mixture of Experts based Multi-task Supervised Learning-from-Crowds), a novel paradigm that treats each worker as a distinct task. By employing a Mixture of Experts (MoE) architecture, MMLC models worker behavior at the item-feature level and projects workers into a "spectral space" to identify ideal "oracle" behavior. Experimental results on image, text, and music datasets demonstrate that MMLC significantly improves label accuracy and robustness compared to existing statistical and supervised approaches.
Introduction
The challenge of noisy labels in crowdsourcing stems from the fact that worker accuracy is not uniform. A worker might excel at identifying general categories but struggle with fine-grained distinctions. Existing methods typically rely on weakly supervised statistical models or static confusion matrices, both of which fail to capture how item-specific characteristics influence a worker's judgment. MMLC addresses this by integrating item features directly into the modeling of worker behavior, framing the problem as a multi-task learning challenge where each worker represents a unique task.
Methodology
The MMLC framework consists of three primary components designed to capture the nuances of crowd labeling:
- Expert Module: A collection of neural networks (experts) that learn various patterns of worker behavior based on extracted item features.
- Gate Module: This module maps each worker's unique identity (via one-hot encoding) into a Worker Spectral Space. This geometric representation captures the unique "style" or behavioral signature of each worker.
- Output Module: By weighting the insights from the experts according to the gate module's output, this module predicts the label a specific worker would assign to a given item.
From this architecture, the authors derive two strategies for truth inference:
- MMLC-owf (Oracle Worker Finding): This strategy identifies the "center" of the worker distribution in the spectral space using clustering techniques like Kernel Density Estimation (KDE). This center represents the "ideal" or oracle worker judgment.
- MMLC-df (Data Filling): This approach uses the trained model to predict missing labels in sparse datasets. By creating a dense label matrix, it significantly enhances the performance of traditional aggregation methods like Majority Voting (MV).
Experimental Results
The model was evaluated using three real-world datasets: LabelMe (1,000 images), Text (1,594 sentences), and Music (700 compositions). MMLC-owf consistently achieved superior accuracy, particularly on the Text (74.31%) and Music (79.14%) datasets, outperforming competitive baselines like CoNAL and CrowdAR.
A key finding is the model's ability to boost existing methods through data filling. On the Music dataset, MMLC-df improved the accuracy of standard Majority Voting from 71.42% to 78.58%. Furthermore, as shown in the redundancy analysis, the model demonstrates strong robustness; while performance improves with more labels per item, MMLC maintains a competitive edge even in low-redundancy scenarios by effectively sharing knowledge between similar workers in the spectral space.
Conclusion
By framing crowdsourcing as a multi-task learning problem, MMLC effectively captures feature-level nuances in worker behavior. While the model shows sensitivity to clustering methods in the spectral space and requires sufficient data to avoid underfitting in extremely low-redundancy cases, it represents a significant step forward in robust truth inference. Future directions include exploring more effective clustering methods and integrating the model into active learning frameworks for optimized worker-item assignment.
Summary updated