Learning deep networks with crowdsourcing for relevance evaluation

Ming Wu, XiaoChun Yin, Qianmu Li, Jing Zhang, Xinqi Feng, Qi Cao, Haiyuan ShenEURASIP J. Wirel. Commun. Netw 2020 · DOI

Introduction

Relevance evaluation—the process of determining how well a search result matches a user's query—is the backbone of Information Retrieval (IR) systems. While expert evaluations provide high-quality data, they are often prohibitively expensive and slow. Crowdsourcing platforms like Amazon Mechanical Turk offer a scalable alternative, but they introduce a significant challenge: noise. Crowd workers possess varying levels of expertise, biases, and attention spans, leading to inconsistent labeling.

The Challenge of Noisy Labels

Traditionally, researchers address crowdsourcing noise through a two-step process. First, they aggregate multiple worker labels using algorithms like Majority Voting or David-Skene to estimate a "ground truth." Second, they train a predictive model on these estimates. This paper argues that this separation is sub-optimal because the aggregation step ignores the inherent features of the data (text and images) and the specific reliability profiles of individual workers.

The proposed framework for relevance evaluation using deep learning in crowdsourcing.

Proposed End-to-End Framework

The authors introduce a multi-modal deep learning architecture that integrates label aggregation directly into the model training process. This end-to-end approach allows the network to learn from the data features and worker reliability simultaneously, effectively "de-noising" the input during backpropagation.

Multi-Modal Feature Extraction

The model processes two distinct types of input to determine relevance:

  1. Images: Features are extracted using a pre-trained VGG-16 model. To ensure these features are relevant to the specific retrieval task, the authors fine-tuned the model using an autoencoder reconstruction approach.
  2. Query Text: Word2Vec embeddings are passed through a Long Short-Term Memory (LSTM) network to capture the semantic context of the search query.

Feature Fusion and the Relevancy Factor

A key technical innovation is the use of a Similarity Matrix ($M_r$). Rather than simply concatenating text and image vectors, the model learns a "relevancy factor" that captures the deep internal relationship between the query and the image. This fused representation is then passed through hidden layers to produce a relevance prediction.

Detailed neural network architecture featuring the CrowdLayer and Relevancy Factor.

The CrowdLayer

The core of the framework is the CrowdLayer, positioned atop the standard output layer. This layer models the "confusion matrix" for each individual worker, representing their probability of providing a specific label given the true class. During training, the model learns to down-weight unreliable workers. For final prediction on new data, this layer is removed, leaving a robust relevance classifier that has been trained on cleaned signals.

Experimental Results

The framework was evaluated using the Div400 dataset, containing 15,871 Flickr photos and 160 queries. To simulate a realistic crowdsourcing environment, the authors modeled five workers with varying sensitivity and specificity levels.

  • Inference Accuracy: The model achieved 93.6% accuracy in identifying true labels from noisy input, significantly outperforming Majority Voting (84.4%) and the GLAD algorithm (90.9%).
  • Generalization: On entirely new query-image pairs, the model reached 73.9% accuracy.
  • Regularization Effect: Remarkably, the model outperformed a standard deep network trained on actual ground-truth labels (71.2%). This suggests that the CrowdLayer acts as a form of regularization, helping the model generalize better by learning from the distribution of worker errors rather than overfitting to a single set of labels.

Conclusion

By treating crowd workers as learnable parameters within a deep neural network, this framework effectively bridges the gap between raw, noisy crowdsourced data and high-performance IR models. The results demonstrate that end-to-end learning is superior to traditional two-step aggregation, particularly when dealing with complex multi-modal data like images and text.

Summary updated

BibTeX
@article{wu2020learning,
  author = {Ming Wu and XiaoChun Yin and Qianmu Li and Jing Zhang and Xinqi Feng and Qi Cao and Haiyuan Shen},
  title = {{Learning deep networks with crowdsourcing for relevance evaluation}},
  journal = {EURASIP J. Wirel. Commun. Netw.},
  year = {2020},
  doi = {10.1186/s13638-020-01697-2},
}
Figure