Abstract
Annotating Semantic Role Labeling (SRL) data usually requires expensive linguistic experts because the tasks are too complex for the average crowd worker. This paper introduces a clever three-stage "Filter-Select-Expert" workflow that leverages a surprising insight: while crowd workers struggle to pick the right label from a long list, they are nearly perfect at identifying the wrong ones. By having the crowd filter out the noise first, the authors reduced expert workload by 4x while maintaining 93-95% accuracy.
Introduction
Semantic Role Labeling (SRL) is the process of determining "who did what to whom" in a sentence. It involves identifying predicate senses (e.g., does "run" mean "compete in a race" or "manage a business"?) and argument labels (e.g., is "John" the agent or the destination?).
For years, this has been a bottleneck in NLP. Unlike sentiment analysis, which most people can do intuitively, SRL is linguistically "heavy." Propbank annotators, for instance, usually require days of training. Previous attempts to crowdsource this either resulted in poor quality or still required experts to step in for over 30% of the cases. The core challenge is cognitive load: when a worker is presented with dozens of subtle label options, they get overwhelmed and make mistakes.
The Filter-Select-Expert Workflow
The authors propose a hybrid workflow designed to "funnel" tasks based on difficulty:
- Filter Phase: Instead of asking "What is the correct label?", workers are asked "Which labels are definitely wrong?" They must select at least half of the options as "least likely." This process repeats iteratively until only three or fewer options remain.
- Select Phase: A new set of workers looks at the remaining (filtered) options and tries to pick the winner. Crucially, they are given a "not sure" option. If the crowd reaches a majority agreement and no one clicks "not sure," the label is accepted.
- Expert Phase: Only the "hard" cases—those where workers disagreed, were unsure, or where the filtering process failed—are sent to a linguistic expert.
Because the Filter and Select phases remove the "easy" cases and narrow down the options for the "hard" ones, the expert's job becomes significantly faster and less frequent.
Evaluation and Key Insights
The researchers evaluated their workflow using the English portion of the CoNLL-2009 shared task (Propbank labels). The evaluation included a pilot scale of 200 tasks and a full scale of 2,014 tasks covering 300 sentences.
- The Power of Negative Knowledge: Non-experts are remarkably good at identifying incorrect options. While they might not know the exact linguistic definition of "Arg2," they can easily see that "Location" is a wrong fit for a person's name.
- Crowd Self-Awareness: Workers are surprisingly good at identifying the limits of their own knowledge. By providing a "not sure" button, the authors successfully diverted ambiguous cases to experts before the crowd could produce "noisy" data.
- Complexity Reduction: By reducing the number of options through filtering, the authors lowered the cognitive load. Worker accuracy jumps significantly as the number of options decreases.
Key Results
The workflow achieved 95% accuracy for predicate senses and 93% for argument labels, matching expert-to-expert agreement levels. Most importantly, the system required expert intervention for only 13% of cases, a 4x reduction compared to previous state-of-the-art hybrid workflows.
Conclusion
This "filtering" approach demonstrates that complex linguistic tasks can be crowdsourced effectively by restructuring them to match human cognitive strengths. While current limitations include the reliance on pre-identified text spans, the workflow's success suggests potential applications in other complex NLP tasks like Syntactic Parsing or Abstract Meaning Representation (AMR).
Summary updated