Low Resource Sequence Tagging with Weak Labels

Edwin Simpson, Jonas Pfeiffer, Iryna Gurevych

Abstract

State-of-the-art sequence tagging models typically require large amounts of high-quality labeled data, which is often unavailable in specialized domains. This paper presents a Bayesian approach to sequence tagging that leverages weak labels from out-of-domain models and noisy crowdsourced annotations. By introducing Variational Combined Supervision (VCS) and Bayesian Sequence Combination (BSC-seq), the authors demonstrate a robust framework for training high-performing taggers in low-resource settings.

Introduction

Sequence tagging is a fundamental task in Natural Language Processing (NLP), encompassing applications from Named Entity Recognition (NER) to argumentative structure analysis. While deep learning models have set new benchmarks, their reliance on extensive "gold standard" data poses a significant barrier in niche fields like medical diagnostics or educational assessment. Traditional workarounds, such as transfer learning or crowdsourcing, often struggle with tiny target datasets or high label noise. This research addresses these challenges by treating various label sources as an ensemble of weak annotators and learning to aggregate them effectively.

Methodology

The core of the proposed approach lies in two key innovations: BSC-seq and VCS.

Bayesian Sequence Combination (BSC-seq)

Building on previous Bayesian aggregation methods, BSC-seq models the sequential dependencies inherent in tagging tasks (e.g., IOB2 encoding). Unlike token-independent models, BSC-seq functions as a Hidden Markov Model (HMM) that learns the reliability and specific biases of each annotator or source model. This allows the system to better handle the structural constraints of sequence labels.

Variational Combined Supervision (VCS)

VCS moves beyond simple two-step pipeline approaches (aggregation followed by training). It integrates the training of a "black-box" sequence tagger—such as a BiLSTM-CRF—directly into the Bayesian inference process. The tagger is treated as an additional member of the ensemble. As the system's estimate of the true labels improves through Variational Bayes (VB) inference, the tagger's parameters are iteratively updated, allowing it to contribute to and learn from the evolving consensus.

Experimental Setup

The framework was evaluated on two primary scenarios:

  1. Domain Transfer (FAMULUS): Identifying argumentative structures in German medical (Med) and teacher education (TEd) diagnostic reasoning datasets.
  2. Crowdsourced Benchmarks: Standard NER (CoNLL 2003) and PICO (medical abstracts) datasets using noisy labels from hundreds of crowd workers.

Key Results

The experimental results highlight the advantages of end-to-end Bayesian integration. In the FAMULUS domain transfer task, combining multiple out-of-domain models via BSC-seq consistently outperformed individual models and simple majority voting.

Table 1: Performance comparison (F1 score) on Med and TEd datasets across different setups.

As shown in Table 1, the BSC-seq+VCS approach (using a DNN or CRF base model) achieves competitive or superior performance compared to in-domain models, particularly in out-of-domain scenarios. For instance, on the Med dataset, BSC-seq with a DNN base model reached an F1 of 49.9, significantly higher than the single-domain CRF baseline of 20.7.

Furthermore, the model demonstrates rapid adaptation capabilities. When small amounts of gold-standard data are introduced, the performance of the ensemble improves sharply.

Figure 1: F1 scores across different categories as a function of additional gold-standard data.

Figure 1 illustrates this adaptation across four categories in the FAMULUS dataset (HG, EG, EE, DC). BSC-seq (orange) consistently maintains a high F1 score, often outperforming standard aggregation methods like Majority Voting (MV) and IBCC, and approaching or exceeding the performance of models trained on all available domains.

On crowdsourced benchmarks, the method also excelled. For NER, BSC-seq+VCS achieved an F1 of 78.0, surpassing the best individual worker (67.3) and the HMM-crowd baseline (74.6). Similarly, it reached 77.5 F1 on the PICO dataset, a substantial improvement over pipeline-based methods (71.2 F1).

Conclusion

This work demonstrates that a Bayesian approach to combining weak labels can effectively bridge the gap in low-resource sequence tagging. By modeling sequential dependencies and integrating model training with label aggregation, VCS provides a flexible, high-performance solution for domains where expert data is scarce. Future work will explore applying this framework to more complex tasks like argument mining and active learning to further reduce annotation costs.

Summary updated

Abstract

Current methods for sequence tagging depend on large quantities of domain-specific training data, limiting their use in new, user-defined tasks with few or no annotations. While crowdsourcing can be a cheap source of labels, it often introduces errors that degrade the performance of models trained on such crowdsourced data. Another solution is to use transfer learning to tackle low resource sequence labelling, but current approaches rely heavily on similar high resource datasets in different languages. In this paper, we propose a domain adaptation method using Bayesian sequence combination to exploit pre-trained models and unreliable crowdsourced data that does not require high resource data in a different language. Our method boosts performance by learning the relationship between each labeller and the target task and trains a sequence labeller on the target domain with little or no gold-standard data. We apply our approach to labelling diagnostic classes in medical and educational case studies, showing that the model achieves strong performance though zero-shot transfer learning and is more effective than alternative ensemble methods. Using NER and information extraction tasks, we show how our approach can train a model directly from crowdsourced labels, outperforming pipeline approaches that first aggregate the crowdsourced data, then train on the aggregated labels.
BibTeX
@inproceedings{simpson2020low,
  author = {Edwin Simpson and Jonas Pfeiffer and Iryna Gurevych},
  title = {{Low Resource Sequence Tagging with Weak Labels}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6415},
}
Figure