Abstract
This paper presents a committee-based active learning (AL) framework designed to reduce the high cost of manual data labeling in biomedical text mining. By combining a state-of-the-art event extraction system with a statistical model that incorporates named entity recognition (NER) and word similarity, the proposed method identifies the most "informative" documents for human experts to annotate. Evaluated against BioNLP Shared Tasks datasets, the method achieves superior performance compared to entropy-based and Gibbs error-based methods. The incorporation of NER and unknown word handling (via word2vec) further enhances the efficiency of document selection for both event extraction and named entity recognition tasks.
Introduction
In the biomedical domain, extracting complex "events"—such as protein-protein interactions or gene regulations—is essential for managing the vast scientific literature. However, training supervised machine learning models for these tasks requires massive amounts of labeled data, which is slow and expensive to produce. Active Learning (AL) aims to mitigate this by allowing the machine to "choose" which unlabeled documents would be most beneficial to learn from next.
While AL has been successful for simpler tasks like identifying gene names, complex event extraction poses unique challenges. Traditional AL methods often rely on unreliable confidence scores or ignore the presence of essential event participants (like proteins). This paper introduces a novel framework that targets these blind spots by focusing on potential false negatives and leveraging semantic context.
Methods
Committee-Based Framework
The core of the proposed method is a committee-based active learning approach that utilizes two distinct components to evaluate unlabeled documents: the Turku Event Extraction System (TEES) and a custom statistical model.
As shown in the flowchart above, the system first uses TEES to filter out documents. If TEES already identifies an event in a sentence, that sentence is deemed "low informativity" because the model already demonstrates proficiency in that area. The system instead prioritizes documents where the statistical model predicts an event exists, but TEES has missed it—effectively targeting the model's "blind spots."
Statistical Ranking and NER Integration
The "ranker" component uses a language model to calculate the probability that a document contains an event based on n-grams or predicate-argument structures. To determine the likelihood of an event expression, the model utilizes a contingency table to correlate the presence of specific N-grams with the occurrence of events.
To further refine selection, the system incorporates two key enhancements:
- NER Integration: Using the Gimli gene/protein recognition system, the model checks for the presence of necessary "actors." If a sentence contains the required entities for an event but the extractor has not yet labeled an event there, it is flagged as highly informative.
- Unknown Word Handling: To address out-of-vocabulary (OOV) words, the authors utilize word2vec to find semantically similar known words. This allows the system to estimate the importance of a document even when it contains terminology not seen during initial training.
Key Results
The researchers used a metric called Deficiency, where a value lower than 1.0 indicates performance superior to random document selection.
- Superiority over Baselines: The proposed method consistently outperformed standard AL techniques like Entropy-based and Gibbs error-based selection across all BioNLP tasks. In the GRO’13 task, the informativity-based method showed an 8.65% improvement over conventional committee methods.
- NER Performance: When adapted for Named Entity Recognition, the method achieved a deficiency score of 0.575 on the CoNLL dataset—a 42% improvement over random selection.
- Ablation Success: Adding NER and word similarity features consistently lowered the deficiency score, validating that these semantic features are vital for high-performance active learning in complex domains.
Conclusion
This research demonstrates that model confidence is a poor proxy for informativity in complex biomedical event extraction. By focusing on potential false negatives and incorporating entity context and semantic bridges for unknown words, the proposed active learning framework significantly reduces the manual annotation effort required to build high-performing extraction systems. Future work will focus on addressing the computational overhead of word similarity calculations and exploring "cold-start" scenarios where no initial seed data is available.
Summary updated