TL;DR
This paper tackles the scalability problem of manually curating "block lists" for dictionary-based Named Entity Recognition (NER) in biomedicine. They introduce a novel method using deep learning (BioBERT) to automatically identify problematic terms, significantly expanding these lists and improving the precision of biomedical text mining systems like STRING, with minimal impact on recall.
What Problem Does This Paper Address?
Biomedical text mining relies heavily on Named Entity Recognition (NER) to identify and classify terms like genes, diseases, and chemicals. Dictionary-based NER systems, while efficient, require curated "block lists" to filter out ambiguous or incorrect matches that can lead to false positives. Traditionally, these block lists are built through time-consuming manual inspection of frequent terms that cause errors. This manual process is a bottleneck, especially when adapting NER systems to new entity types or domains, as it doesn't scale well with the ever-growing volume of biomedical literature. The authors aim to automate this crucial block list generation process to improve the precision and scalability of dictionary-based NER.
Methods
The core of the proposed method involves leveraging deep learning to enhance dictionary-based NER. Here's a breakdown:
- Consensus Dataset Generation: The authors first created a large, high-quality dataset by comparing the NER outputs of three established biomedical NER tools: PubTator, EVEX, and their own dictionary-based tagger (JensenLab tagger). They identified millions of text spans where all three tools agreed on both the entity boundaries and types. This consensus served as a reliable source of ground truth.
- Contextual Feature Engineering: From this consensus dataset, they generated positive and negative examples of contexts for four key entity types: genes/gene products, diseases, species, and chemicals. Crucially, they masked the actual entity mentions, forcing the model to learn from the surrounding text.
- Deep Learning Model Training: A Transformer-based model, specifically BioBERT (a BERT model pre-trained on biomedical text), was trained as a multi-class entity type classifier on these contextual examples. This model learns to predict the entity type based solely on the context.
- Automated Block List Generation: The trained BioBERT model was then used to predict entity types for all matches identified by the JensenLab tagger (using only its manually curated block lists). By analyzing the model's confidence scores (specifically, the probability of a match not belonging to the tagger-assigned class), they could identify problematic names that were likely to be misclassified. This process allowed for the generation of both corpus-wide (global) and document-specific (local) block lists.
Datasets
- Consensus Dataset: A newly generated dataset comprising over 12.5 million text spans where three different NER methods agreed on entity boundaries and types. This dataset was split into a 125k training set and a 12.5M training set.
- PubMed Abstracts and PMC BioC Collection: These were used as the primary corpus for running the tagger and generating the consensus dataset.
- KEGG and DISEASES databases: Used for evaluating the impact of the improved NER system on downstream relation extraction tasks.
- S1000 corpus: Used for evaluating the effect of the new block list on species NER.
Key Insights and Ideas
- Deep Learning as a Contextual Filter: The paper demonstrates that deep learning models, even when trained on masked entity mentions, can effectively learn contextual cues to identify problematic terms that would otherwise be misclassified by dictionary-based systems.
- Automated Block List Generation is Scalable: The manual process of creating block lists is a major bottleneck. This work shows that an automated approach, powered by deep learning, can scale to identify a vast number of problematic terms, significantly expanding the utility of dictionary-based NER.
- Document-Specific Blocking is Crucial for Ambiguity: The introduction of document-specific block lists is a key innovation. It allows for more nuanced filtering, addressing cases where a term might be correct in one context but erroneous in another (e.g., "wingless" as a protein vs. an insect).
- Precision Gains Outweigh Recall Loss: While adding automatically generated block lists inevitably leads to a slight decrease in recall (as some correct matches might be blocked), the significant gains in precision are shown to be more impactful for the overall utility of the system.
Key Results
- BioBERT Model Performance: The trained BioBERT model achieved an impressive F1-score of 96.7% on the 12.5M dataset for entity type classification.
- Block List Expansion: The automated method effectively doubled the size of the corpus-wide block list.
- Precision Improvement: Text mining precision increased by approximately 5.5% on average, with notable improvements of over 8.5% for chemical names and 7.5% for gene names.
- Recall Impact: A minor drop in recall (0.6%) was observed, which was deemed acceptable given the precision gains.
- Downstream Impact: The improvements positively affected several biological databases, including the widely used STRING database.
Limitations and Future Work
- Manual Threshold Tuning: While the block list generation is automated, the authors still had to manually inspect lists to determine the optimal thresholds for inclusion in both global and local lists.
- Quality of Local Lists: The paper notes that generating high-quality local allow/block lists was not possible for all entity types (specifically diseases and species), suggesting further work might be needed in this area.
- Potential for Further Automation: The authors suggest that future work could explore more automated methods for setting these thresholds, further reducing manual effort.
- Generalizability: While successful in the biomedical domain, the approach could potentially be adapted to other domains with appropriate pre-trained models and domain-specific corpora.
Summary updated