Mixture of Experts Based Multi-Task Supervise Learning from Crowds

Tao Han, Huaixuan Shi, Xinyi Ding, Xiao Ma, Huamao Gu, Yili FangAAAI 2025 · DOI

Abstract

Crowdsourcing provides a cost-effective solution for labeling large-scale datasets, yet it is inherently prone to human error and inconsistency. Traditional truth inference methods often overlook the relationship between item features and worker performance. This paper introduces MMLC (Mixture of Experts based Multi-task Supervised Learning-from-Crowds), a novel paradigm that treats each worker as a distinct task. By employing a Mixture of Experts (MoE) architecture, MMLC models worker behavior at the item-feature level and projects workers into a "spectral space" to identify ideal "oracle" behavior. Experimental results on image, text, and music datasets demonstrate that MMLC significantly improves label accuracy and robustness compared to existing statistical and supervised approaches.

Introduction

The challenge of noisy labels in crowdsourcing stems from the fact that worker accuracy is not uniform. A worker might excel at identifying general categories but struggle with fine-grained distinctions. Existing methods typically rely on weakly supervised statistical models or static confusion matrices, both of which fail to capture how item-specific characteristics influence a worker's judgment. MMLC addresses this by integrating item features directly into the modeling of worker behavior, framing the problem as a multi-task learning challenge where each worker represents a unique task.

Methodology

The MMLC framework consists of three primary components designed to capture the nuances of crowd labeling:

  1. Expert Module: A collection of neural networks (experts) that learn various patterns of worker behavior based on extracted item features.
  2. Gate Module: This module maps each worker's unique identity (via one-hot encoding) into a Worker Spectral Space. This geometric representation captures the unique "style" or behavioral signature of each worker.
  3. Output Module: By weighting the insights from the experts according to the gate module's output, this module predicts the label a specific worker would assign to a given item.
The MMLC architecture featuring Expert, Gate, and Output modules, alongside the Worker Spectral Space.

From this architecture, the authors derive two strategies for truth inference:

  • MMLC-owf (Oracle Worker Finding): This strategy identifies the "center" of the worker distribution in the spectral space using clustering techniques like Kernel Density Estimation (KDE). This center represents the "ideal" or oracle worker judgment.
  • MMLC-df (Data Filling): This approach uses the trained model to predict missing labels in sparse datasets. By creating a dense label matrix, it significantly enhances the performance of traditional aggregation methods like Majority Voting (MV).

Experimental Results

The model was evaluated using three real-world datasets: LabelMe (1,000 images), Text (1,594 sentences), and Music (700 compositions). MMLC-owf consistently achieved superior accuracy, particularly on the Text (74.31%) and Music (79.14%) datasets, outperforming competitive baselines like CoNAL and CrowdAR.

Accuracy vs. Redundancy across three datasets, showing MMLC-owf's performance relative to baseline methods.

A key finding is the model's ability to boost existing methods through data filling. On the Music dataset, MMLC-df improved the accuracy of standard Majority Voting from 71.42% to 78.58%. Furthermore, as shown in the redundancy analysis, the model demonstrates strong robustness; while performance improves with more labels per item, MMLC maintains a competitive edge even in low-redundancy scenarios by effectively sharing knowledge between similar workers in the spectral space.

Conclusion

By framing crowdsourcing as a multi-task learning problem, MMLC effectively captures feature-level nuances in worker behavior. While the model shows sensitivity to clustering methods in the spectral space and requires sufficient data to avoid underfitting in extremely low-redundancy cases, it represents a significant step forward in robust truth inference. Future directions include exploring more effective clustering methods and integrating the model into active learning frameworks for optimized worker-item assignment.

Summary updated

BibTeX
@inproceedings{han2025mixture,
  author = {Tao Han and Huaixuan Shi and Xinyi Ding and Xiao Ma and Huamao Gu and Yili Fang},
  title = {{Mixture of Experts Based Multi-Task Supervise Learning from Crowds}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i13.33561},
}
Figure