A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels

Youxuan Jiang, Huaiyu Zhu, Jonathan K. Kummerfeld, Yunyao Li, Walter S. LaseckiEMNLP 2020 · OpenReview

Abstract

Annotating Semantic Role Labeling (SRL) data usually requires expensive linguistic experts because the tasks are too complex for the average crowd worker. This paper introduces a clever three-stage "Filter-Select-Expert" workflow that leverages a surprising insight: while crowd workers struggle to pick the right label from a long list, they are nearly perfect at identifying the wrong ones. By having the crowd filter out the noise first, the authors reduced expert workload by 4x while maintaining 93-95% accuracy.

Introduction

Semantic Role Labeling (SRL) is the process of determining "who did what to whom" in a sentence. It involves identifying predicate senses (e.g., does "run" mean "compete in a race" or "manage a business"?) and argument labels (e.g., is "John" the agent or the destination?).

For years, this has been a bottleneck in NLP. Unlike sentiment analysis, which most people can do intuitively, SRL is linguistically "heavy." Propbank annotators, for instance, usually require days of training. Previous attempts to crowdsource this either resulted in poor quality or still required experts to step in for over 30% of the cases. The core challenge is cognitive load: when a worker is presented with dozens of subtle label options, they get overwhelmed and make mistakes.

The Filter-Select-Expert Workflow

The authors propose a hybrid workflow designed to "funnel" tasks based on difficulty:

  1. Filter Phase: Instead of asking "What is the correct label?", workers are asked "Which labels are definitely wrong?" They must select at least half of the options as "least likely." This process repeats iteratively until only three or fewer options remain.
  2. Select Phase: A new set of workers looks at the remaining (filtered) options and tries to pick the winner. Crucially, they are given a "not sure" option. If the crowd reaches a majority agreement and no one clicks "not sure," the label is accepted.
  3. Expert Phase: Only the "hard" cases—those where workers disagreed, were unsure, or where the filtering process failed—are sent to a linguistic expert.
The annotation interface for the Filter phase, where workers identify incorrect labels to reduce cognitive load.

Because the Filter and Select phases remove the "easy" cases and narrow down the options for the "hard" ones, the expert's job becomes significantly faster and less frequent.

Evaluation and Key Insights

The researchers evaluated their workflow using the English portion of the CoNLL-2009 shared task (Propbank labels). The evaluation included a pilot scale of 200 tasks and a full scale of 2,014 tasks covering 300 sentences.

  • The Power of Negative Knowledge: Non-experts are remarkably good at identifying incorrect options. While they might not know the exact linguistic definition of "Arg2," they can easily see that "Location" is a wrong fit for a person's name.
  • Crowd Self-Awareness: Workers are surprisingly good at identifying the limits of their own knowledge. By providing a "not sure" button, the authors successfully diverted ambiguous cases to experts before the crowd could produce "noisy" data.
  • Complexity Reduction: By reducing the number of options through filtering, the authors lowered the cognitive load. Worker accuracy jumps significantly as the number of options decreases.

Key Results

The workflow achieved 95% accuracy for predicate senses and 93% for argument labels, matching expert-to-expert agreement levels. Most importantly, the system required expert intervention for only 13% of cases, a 4x reduction compared to previous state-of-the-art hybrid workflows.

Table 3: Performance comparison showing that the proposed workflow achieves high accuracy with significantly lower expert involvement.

Conclusion

This "filtering" approach demonstrates that complex linguistic tasks can be crowdsourced effectively by restructuring them to match human cognitive strengths. While current limitations include the reliance on pre-identified text spans, the workflow's success suggests potential applications in other complex NLP tasks like Syntactic Parsing or Abstract Meaning Representation (AMR).

Summary updated

BibTeX
@inproceedings{jiang2020novel,
  author = {Youxuan Jiang and Huaiyu Zhu and Jonathan K. Kummerfeld and Yunyao Li and Walter S. Lasecki},
  title = {{A Novel Workflow for Accurately and Efficiently Crowdsourcing Predicate Senses and Argument Labels}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=MDqi9GjogZM},
}
Figure