Abstract
Crowdsourcing is essential for generating large-scale labeled datasets, yet it often suffers from low data quality due to worker inexperience. This paper proposes ILPCE (Interactive Learning with Proactive Cognition Enhancement), a framework that shifts the paradigm from treating workers as static sensors to active learners. By integrating "machine teaching" based on psychological models of human categorization, ILPCE proactively improves worker reliability during the labeling task. Experimental results on fine-grained image classification tasks demonstrate that ILPCE significantly boosts both worker accuracy and the final machine learning model's performance compared to state-of-the-art methods.
Introduction
The success of supervised learning relies heavily on high-quality labeled data. In crowdsourcing environments, workers often lack the domain expertise required for specialized tasks, such as distinguishing between similar species of birds or butterflies. Traditional solutions—such as repeated labeling for consensus or active learning to select informative samples—often fail to address the root cause: the instability and potential incompetence of the human labelers.
The authors identify a critical gap in current methodologies: active learning curves for crowdsourced data are often jagged and slow because worker reliability is treated as a fixed or slowly varying parameter. ILPCE aims to create a "smoothly rising learning curve" by enhancing worker cognition in real-time through an interactive teaching loop.
The ILPCE Framework
The ILPCE framework consists of three integrated components that form a virtuous cycle of intelligence enhancement.
- Bayesian Truth Inference Model (BIM): This probabilistic model simultaneously estimates the true label of an instance, the difficulty of the task, and the reliability of each worker. Unlike simpler models, BIM treats task difficulty as a continuous variable that modulates worker performance.
- Active Learning Strategy: The system selects query instances based on three criteria: label uncertainty (disagreement among workers), model uncertainty (AI confidence), and representativeness (how well the sample covers the data distribution).
- Machine Teaching: This is the core innovation. Utilizing the Generalized Context Model (GCM)—a psychological model of human categorization—the system selects "exemplars" to present to the worker alongside the query instance. These exemplars are mathematically optimized to help the worker learn the boundaries between different classes, effectively training them on the job.
Experimental Evaluation
The framework was evaluated on three challenging real-world image classification datasets requiring domain expertise:
- Butterflies: 1,000 images across 5 species.
- Birds: 400 images across 8 species.
- Dogs: 1,420 images across 10 breeds.
These tasks require fine-grained visual discrimination, making them ideal for testing whether proactive teaching can bridge the expertise gap.
Key Results
ILPCE was compared against several state-of-the-art methods, including AMR and IMPACT. The results consistently showed that ILPCE outperforms existing approaches in both model accuracy and worker proficiency.
- Model Accuracy: ILPCE achieved significantly higher prediction accuracy. For instance, on the Butterflies dataset, it outperformed the baseline AMR method by approximately 15%.
- Worker Reliability: Average worker accuracy improved steadily throughout the tasks. In the Butterflies experiment, worker accuracy rose by nearly 9 percentage points as they interacted with the system's exemplars.
- Efficiency and Speed: While the "learning phase" initially increased the time spent per task, workers using ILPCE eventually showed the best execution speed acceleration as their proficiency grew, proving the long-term efficiency of the teaching approach.
Conclusion
ILPCE demonstrates that machine intelligence can be leveraged not just to process human output, but to actively enhance human capability. By grounding AI strategies in psychological models of cognition, the framework creates a co-evolutionary loop where better teaching leads to better labels, which in turn leads to better models. Future work may explore personalized teaching strategies that adapt to individual worker cognition parameters.
Summary updated