Abstract
State-of-the-art sequence tagging models typically require large amounts of high-quality labeled data, which is often unavailable in specialized domains. This paper presents a Bayesian approach to sequence tagging that leverages weak labels from out-of-domain models and noisy crowdsourced annotations. By introducing Variational Combined Supervision (VCS) and Bayesian Sequence Combination (BSC-seq), the authors demonstrate a robust framework for training high-performing taggers in low-resource settings.
Introduction
Sequence tagging is a fundamental task in Natural Language Processing (NLP), encompassing applications from Named Entity Recognition (NER) to argumentative structure analysis. While deep learning models have set new benchmarks, their reliance on extensive "gold standard" data poses a significant barrier in niche fields like medical diagnostics or educational assessment. Traditional workarounds, such as transfer learning or crowdsourcing, often struggle with tiny target datasets or high label noise. This research addresses these challenges by treating various label sources as an ensemble of weak annotators and learning to aggregate them effectively.
Methodology
The core of the proposed approach lies in two key innovations: BSC-seq and VCS.
Bayesian Sequence Combination (BSC-seq)
Building on previous Bayesian aggregation methods, BSC-seq models the sequential dependencies inherent in tagging tasks (e.g., IOB2 encoding). Unlike token-independent models, BSC-seq functions as a Hidden Markov Model (HMM) that learns the reliability and specific biases of each annotator or source model. This allows the system to better handle the structural constraints of sequence labels.
Variational Combined Supervision (VCS)
VCS moves beyond simple two-step pipeline approaches (aggregation followed by training). It integrates the training of a "black-box" sequence tagger—such as a BiLSTM-CRF—directly into the Bayesian inference process. The tagger is treated as an additional member of the ensemble. As the system's estimate of the true labels improves through Variational Bayes (VB) inference, the tagger's parameters are iteratively updated, allowing it to contribute to and learn from the evolving consensus.
Experimental Setup
The framework was evaluated on two primary scenarios:
- Domain Transfer (FAMULUS): Identifying argumentative structures in German medical (Med) and teacher education (TEd) diagnostic reasoning datasets.
- Crowdsourced Benchmarks: Standard NER (CoNLL 2003) and PICO (medical abstracts) datasets using noisy labels from hundreds of crowd workers.
Key Results
The experimental results highlight the advantages of end-to-end Bayesian integration. In the FAMULUS domain transfer task, combining multiple out-of-domain models via BSC-seq consistently outperformed individual models and simple majority voting.
As shown in Table 1, the BSC-seq+VCS approach (using a DNN or CRF base model) achieves competitive or superior performance compared to in-domain models, particularly in out-of-domain scenarios. For instance, on the Med dataset, BSC-seq with a DNN base model reached an F1 of 49.9, significantly higher than the single-domain CRF baseline of 20.7.
Furthermore, the model demonstrates rapid adaptation capabilities. When small amounts of gold-standard data are introduced, the performance of the ensemble improves sharply.
Figure 1 illustrates this adaptation across four categories in the FAMULUS dataset (HG, EG, EE, DC). BSC-seq (orange) consistently maintains a high F1 score, often outperforming standard aggregation methods like Majority Voting (MV) and IBCC, and approaching or exceeding the performance of models trained on all available domains.
On crowdsourced benchmarks, the method also excelled. For NER, BSC-seq+VCS achieved an F1 of 78.0, surpassing the best individual worker (67.3) and the HMM-crowd baseline (74.6). Similarly, it reached 77.5 F1 on the PICO dataset, a substantial improvement over pipeline-based methods (71.2 F1).
Conclusion
This work demonstrates that a Bayesian approach to combining weak labels can effectively bridge the gap in low-resource sequence tagging. By modeling sequential dependencies and integrating model training with label aggregation, VCS provides a flexible, high-performance solution for domains where expert data is scarce. Future work will explore applying this framework to more complex tasks like argument mining and active learning to further reduce annotation costs.
Summary updated