Introduction
Crowdsourcing is a widely used method for labeling large datasets, but it often relies on the assumption that the majority of workers are correct. This assumption fails in specialized or technical domains where average workers lack expertise, leading to noisy or incorrect labels. While transfer learning has been proposed to mitigate this by using data from related domains, it typically requires large amounts of labeled source data, which is often as difficult to obtain as the target labels themselves.
The CrowdMeta Framework
To address these challenges, we propose CrowdMeta, a truth inference model that leverages meta-learning to transfer knowledge from related domains using very few examples. Unlike traditional transfer learning that focuses on simple feature copying, CrowdMeta learns high-level "meta-knowledge" about data representation.
The framework consists of three key components:
- Meta-Knowledge Transfer: Using a few-shot meta-learning approach (inspired by MAML), the model is pre-trained to learn high-level representations across different tasks.
- Fine-Tuning: With these meta-features, the model requires only a small number of instances ($K$) from the target domain to adapt effectively.
- Truth Inference via EM: The model extracts a high-level representation ($V$) for each task and employs an Expectation-Maximization (EM) algorithm to jointly estimate the true label, worker expertise ($e^w$), and label quality ($q^w_i$).
Experimental Evaluation
We evaluated CrowdMeta using a simulation based on the MiniImagenet dataset. The target tasks included 500 randomly selected instances, with 30 simulated workers ranging from spammers to experts. We compared CrowdMeta against Majority Vote (MV) and existing transfer-learning models: CrowdTrU and CrowdMKT.
Key Results
The experiments focused on the model's performance under varying data constraints. In the low-data scenario (Experiment 1), where all models were limited to 5 instances per class in the source domain, CrowdMeta achieved 72.8% accuracy. In the high-data scenario (Experiment 2), even when competing models were given 12 times more data (60 instances per class), CrowdMeta maintained its lead with 73.2% accuracy while still only using its original 5 instances.
Conclusion
CrowdMeta demonstrates that meta-learning allows for effective knowledge transfer with significantly less data than traditional methods. By focusing on high-level representations, the model remains robust to noise and accurately distinguishes worker expertise, even when the majority of crowdsourced labels are incorrect. Future work will explore the integration of active learning to further optimize task assignment and minimize costs.
Summary updated