Crowdsourced Semantic Matching of Multi-Label Annotations

Lei Duan, Satoshi Oyama, Masahito Kurihara, Haruhiko SatoIJCAI 2015

Introduction

In fields such as emotion recognition and text categorization, the absence of a universal "gold standard" taxonomy often leads to a "Taxonomy Tower of Babel." Different research projects may employ distinct label sets—for instance, mapping a sentence to Ekman’s six basic emotions versus Nakamura’s ten categories. This lack of interoperability hinders the reuse of datasets and models across projects.

Bridging this gap typically involves learning a mapping function between taxonomies. However, in multi-label environments, the complexity is significant: the number of possible label combinations grows exponentially ($2^N$). A direct mapping approach, such as Joint Maximum Likelihood Estimation (J-MLE), requires an impractical volume of training data to cover all potential combinations. Furthermore, when relying on crowdsourced annotations to establish these mappings, worker noise and bias can severely degrade the quality of the learned function.

The Cascaded Method

To address these challenges, this paper proposes a Cascaded Method for multi-label translation. Instead of treating entire label sets as atomic units, the method decomposes the problem into relationships between individual labels through a latent semantic space.

Comparison of (a) direct mapping and (b) the proposed cascaded mapping approach.

The approach consists of three primary components:

  1. Decomposition: By assuming statistical independence between labels, the method breaks down the mapping of a source vector $s$ to a target vector $t$ into estimating the probability of individual target labels given individual source labels. This reduces the problem's complexity from exponential to linear relative to the number of labels.
  2. Probabilistic Mapping: Utilizing Bayes’ theorem, the system establishes a matching function that calculates the posterior probability of a target label set based on the observed source label set.
  3. Annotator Modeling (C-DS): To mitigate crowdsourcing noise, the authors integrate the Dawid-Skene model. This treats the true label as a latent variable and estimates the "competence" (confusion matrix) of each worker. This allows the system to weight contributions from reliable workers more heavily, improving overall translation accuracy.

Experimental Setup

The proposed method was evaluated using two real-world datasets of Japanese children's narratives:

  • "Love" (Although we are in love): 63 sentences.
  • "Apple" (Little Masa and a red apple): 78 sentences.

The experiments involved mapping between Ekman’s Taxonomy (6 labels: Sadness, Anger, Happiness, Disgust, Surprise, Fear) and Nakamura’s Taxonomy (10 labels: Relief, Anger, Sadness, Happiness, Fondness, Excitement, Disgust, Surprise, Fear, Shame). Annotations were gathered via the Lancers crowdsourcing platform, with 30 workers per sentence to establish a robust baseline for evaluation.

Results and Discussion

The evaluation compared three models: the baseline J-MLE, the Cascaded MLE (C-MLE), and the Cascaded Dawid-Skene (C-DS).

Average accuracy of J-MLE, C-MLE, and C-DS across different narratives and mapping directions as a function of the number of annotators.

Key Findings

  • Superior Performance: The cascaded models (C-MLE and C-DS) consistently outperformed the standard J-MLE across all experimental conditions.
  • Efficiency and Scalability: The C-DS model achieved over 90% accuracy with as few as five annotators per instance. This suggests that high-quality taxonomy mappings can be constructed cost-effectively.
  • Robustness to Data Sparsity: Unlike J-MLE, which fails when encountering unseen label combinations in the test set (the "zero-shot" problem), the cascaded approach handles these cases gracefully by leveraging the learned relationships of constituent labels.
  • Asymmetric Mapping Accuracy: Accuracy was generally higher when mapping from a more granular taxonomy (Nakamura’s 10 labels) to a sparser one (Ekman’s 6 labels). The richer semantic information in the source taxonomy facilitates more precise target label estimation.

Conclusion and Future Work

The cascaded probabilistic method provides a robust framework for translating multi-label annotations between disparate taxonomies. By decomposing label sets and modeling annotator reliability, the approach achieves high accuracy with minimal crowdsourced input.

While effective, the current model's assumption of label independence is a simplification; emotions like "Joy" and "Relief" are often correlated. Future research will focus on incorporating these dependencies and testing the method's performance on larger, web-scale datasets. Additionally, automating the discovery of label correlations during the estimation process remains a key objective for enhancing the model's sophistication.

Summary updated

BibTeX
@inproceedings{duan2015crowdsourced,
  author = {Lei Duan and Satoshi Oyama and Masahito Kurihara and Haruhiko Sato},
  title = {{Crowdsourced Semantic Matching of Multi-Label Annotations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure