Abstract
Machine learning performance is heavily dependent on high-quality labeled data, yet the process of obtaining these labels from non-expert annotators is often inefficient. This paper introduces Annotation Curricula (AC), a method designed to "implicitly train" annotators by presenting tasks in order of increasing difficulty. By leveraging educational principles, AC reduces cognitive load and total annotation time by up to 31% without compromising data quality, offering a human-centric alternative to traditional active learning strategies.
Introduction
While crowdsourcing and citizen science have become staples for data labeling, non-expert annotators often struggle with a steep learning curve involving complex schemas and domain-specific nuances. Traditional Active Learning (AL) techniques, which prioritize "informative" (and thus difficult or ambiguous) instances for model training, inadvertently increase the cognitive burden on humans. This often leads to higher error rates and slower performance.
This research explores whether changing the sequence of data presentation—moving from easy to hard instances—can facilitate faster learning and more efficient labeling. This concept, termed Annotation Curricula, aims to integrate training directly into the annotation process.
Methods
The authors formalize the AC framework through two primary components: a Strategy for ordering and an Estimator for predicting instance difficulty.
Two types of estimators were evaluated:
- Non-adaptive (Heuristics): These utilize static metrics such as sentence length, readability scores, or Masked Language Modeling (MLM) loss. MLM loss serves as a task-agnostic proxy; if a model finds a sentence difficult to predict, a human is likely to find it difficult to process.
- Adaptive (Interactive): These employ regression models (e.g., Gaussian Processes or GBMs) to predict an individual annotator's speed in real-time based on their historical performance.
Experimental Setup
The approach was validated through two distinct phases: an offline simulation using historical datasets and a controlled user study.
The offline simulation utilized three NLP datasets: Muc7T (Named Entity Recognition), SigIE (Email signatures), and SPEC (Sentence classification). For the user study, a benchmark based on COVIDLies was developed, where 40 volunteers matched tweets to pandemic-related misconceptions. Difficulty was manually controlled by selecting semantically similar (hard) or distinct (easy) distractors.
Key Results and Insights
- Efficiency Gains: The user study demonstrated that the MLM-based curriculum group completed tasks in an average of 1,273 seconds, compared to 1,852 seconds for the random control group—a 31% reduction in time.
- Quality Consistency: Accuracy remained stable at approximately 84-85% across all groups, indicating that the speedup did not come at the expense of label precision.
- MLM as a Difficulty Proxy: A significant finding was the robustness of MLM loss as a task-agnostic difficulty estimator. It consistently outperformed simpler heuristics like sentence length, which proved brittle in specific domains like email signature extraction.
- The Human-AI Conflict: The study highlights a fundamental tension: while Active Learning is optimal for model convergence, it is sub-optimal for human performance. AC successfully flips this priority to favor the annotator's cognitive state.
Limitations and Future Work
While promising, the study noted that adaptive models often face a "cold start" problem, struggling to learn an annotator's behavior within short tasks. Future systems may benefit from a hybrid approach that begins with a heuristic like MLM and transitions to an adaptive model as more data is collected. Additionally, further research is required to determine if these benefits scale to expert annotators, who may prioritize thematic grouping over difficulty-based sequencing, or to large-scale crowdsourcing efforts involving thousands of instances.
Summary updated