Leveraging Active Learning to Minimise SRL Annotation Across Corpora

Skatje Myers, Martha PalmerACL 2023

TL;DR

This paper explores how to make Semantic Role Labeling (SRL) annotation more efficient using active learning (AL). They introduce a new predicate-focused AL strategy that significantly speeds up annotation on specialized corpora, though a sentence-level approach remains more effective for the large, diverse OntoNotes dataset.

What Problem Does This Paper Address?

Manually annotating data is crucial for training high-quality Natural Language Processing (NLP) models, but it's expensive and time-consuming, especially for tasks like Semantic Role Labeling (SRL) that require expert knowledge. SRL aims to identify and label the "who, what, and when" of predicates in a sentence, providing rich information for downstream tasks. Active learning (AL) offers a solution by intelligently selecting the most informative data points for annotation, thereby reducing the overall annotation burden. However, existing AL methods for SRL often treat entire sentences as units, which can be inefficient when sentences contain multiple predicates, some of which might be less informative than others. This paper investigates a more granular approach to AL for SRL, focusing on selecting individual predicates.

Methods

The core of this work lies in applying Bayesian Active Learning by Disagreement (BALD) to SRL. BALD is a popular AL strategy that leverages model uncertainty to select data. The authors adapt BALD by using dropout during model inference to approximate a Gaussian process, generating multiple predictions for each instance.

They propose and compare two main selection strategies:

  • BALD SENTENCES: This method, similar to prior work, selects entire sentences based on the average disagreement score across all predicates within that sentence.
  • BALD PREDICATES: This is their novel, more granular approach. Instead of scoring entire sentences, it scores individual predicate-argument structures. It then selects specific predicate instances to add to the training pool. This allows for a finer-grained selection of informative parts of the data.

They also compare these AL strategies against a random predicate selection baseline to gauge the effectiveness of their active learning approaches.

Datasets

The authors evaluate their methods on four distinct corpora:

  • THYME Colon: Clinical notes related to colon cancer treatment, characterized by specialized medical vocabulary and formulaic sentence structures.
  • ClearEarth Earthquakes: Text from Wikipedia and educational sources about earthquakes and plate tectonics, featuring scientific language.
  • ClearEarth Ecology: Text from Wikipedia, educational websites, and glossaries covering genetics, evolution, and food chains.
  • OntoNotes 5.0: A large, multi-genre corpus including news, telephone conversations, weblogs, and Usenet, serving as a broad-purpose benchmark.

Key Insights and Ideas

  • Granularity Matters (Sometimes): The predicate-focused BALD PREDICATES method shows significant early gains on specialized, narrow-domain corpora (THYME Colon, Earthquakes, Ecology). This suggests that focusing on the most uncertain predicates is highly effective when the domain's linguistic patterns are consistent.
  • Diversity is Key for Broad Domains: For the large and diverse OntoNotes corpus, the sentence-level BALD SENTENCES approach outperforms BALD PREDICATES for a substantial portion of the training. This indicates that selecting whole sentences might be better at capturing the broader range of linguistic phenomena and vocabulary present in such a corpus.
  • Polysemy as a Factor: The authors hypothesize that the higher polysemy (multiple meanings) of words in OntoNotes might explain why the predicate-focused approach is slower to yield benefits. More diverse word senses could lead to more varied disagreement scores, making it harder for the predicate-level selection to consistently identify the most informative instances.
  • Disagreement Decreases with Performance: As the model's performance improves and plateaus, the disagreement scores for selected batches generally decrease, suggesting that the model is becoming more confident and the annotation is yielding diminishing returns.

Key Results

  • On the THYME Colon, Ecology, and Earthquakes corpora, BALD PREDICATES consistently outperformed both BALD SENTENCES and random selection after approximately 1,500-2,000 predicates were annotated.
  • For OntoNotes, BALD SENTENCES performed better than BALD PREDICATES until around 7,000 predicates, after which BALD PREDICATES began to catch up.
  • The paper also explored the impact of batch size on performance, finding that THYME Colon was relatively robust to larger batches, while Earthquakes showed a clearer need for smaller batch sizes.
  • Analysis of diversity showed that active learning methods tended to select sentences with higher semantic similarity compared to random selection, which could be wasteful in some cases.

Limitations and Future Work

  • Computational Cost: Active learning, while reducing annotation costs, can be computationally intensive due to the iterative training and selection process. The net savings depend on computing resources and annotator costs.
  • Batch Size Optimization: While they investigated batch sizes, determining the optimal size for different corpora and tasks requires further investigation to balance performance gains with training time.
  • Predicate Selection Diversity: The BALD PREDICATES method sometimes selects multiple instances of the same predicate lemma. Future work could explore limiting the selection batch to unique lemmas to potentially improve efficiency.
  • Sentence-Level Redundancy: For some corpora, like THYME Colon, there's evidence of the active learner choosing redundant sentences. Future work could focus on reducing sentence-level semantic similarity in selected batches.
  • Polysemy Impact: Further investigation is needed to fully understand how polysemy in broad domains affects the effectiveness of predicate-level active learning.

Summary updated

BibTeX
@inproceedings{myers2023leveraging,
  author = {Skatje Myers and Martha Palmer},
  title = {{Leveraging Active Learning to Minimise SRL Annotation Across Corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure