TL;DR
This paper tackles the challenge of extracting structured knowledge from biomedical text, a task hindered by the scarcity of labeled data. They propose a "slot-filling" approach, reframing Information Extraction as a question-answering problem, which significantly reduces the need for task-specific training data and enables zero-shot learning.
What Problem Does This Paper Address?
Extracting structured information (like relationships between entities) from unstructured text is crucial for building knowledge bases. Traditional methods rely on a pipeline of Named Entity Recognition (NER), Entity Linking (EL), and Relation Extraction (RE), each requiring extensive, domain-specific annotated data. This is a major bottleneck, especially in biomedicine, where the sheer variety of entities and relations makes data annotation incredibly expensive and time-consuming. The authors highlight that even established ontologies like UMLS contain a vast number of semantic and relation types.
Methods
The core of the paper's contribution is framing biomedical Information Extraction (IE) as a slot-filling task, analogous to open-domain question answering (QA). Instead of training separate models for NER, EL, and RE, they use a two-stage approach:
- Retrieval: A dense passage retriever (DPR), specifically a dual BERT-based bi-encoder, is trained to find relevant passages from a large corpus (PubMed) given a query (e.g., "drug X treats disease Y"). This replaces the need to scan every sentence for potential relations.
- Reading Comprehension: A Transformer-based reader model (BioBERT) then takes the retrieved passages and the query to extract the "answer" or "tail entity" of the relation.
This approach allows for "zero-shot" settings because it doesn't explicitly look for predefined entity or relation types but rather learns to find answers to specific queries. They also leverage existing pre-trained models and fine-tune them on their newly created biomedical dataset.
Datasets
The authors introduce a new benchmark called BioSF for biomedical slot filling. They construct this dataset by transforming existing biomedical NER and RE datasets (BioCreative V CDR, BioCreative VI ChemProt, DDIExtraction 2013) into a question-answering format. They also use a PubMed dump as their knowledge source. For evaluation, they use the BioSF test set and also Hetionet, a network of biomedical knowledge, for zero-shot evaluation.
Key Insights and Ideas
- Slot-filling as a paradigm shift: Reframing IE as QA significantly alleviates the data scarcity problem in specialized domains like biomedicine.
- In-domain data is crucial for DPR: While general-domain pre-trained DPR models show some promise, they perform poorly compared to BM25 on biomedical data. Fine-tuning DPR on in-domain data (BioSF) leads to substantial improvements, highlighting the importance of domain-specific training.
- Reader performance is highly dependent on training data: Baselines trained on general QA or zero-shot RE datasets perform poorly on biomedical slot filling compared to models trained on in-domain BioSF data, underscoring the value of task-specific datasets.
- Biomedical QA is inherently multi-answer: Unlike general domain QA where a query often has a single answer, biomedical queries frequently have multiple valid answers. This makes evaluation more complex and requires different metrics.
Key Results
- Retrieval: Their custom-trained biomedical DPR model (Bio-DPR) significantly outperforms BM25 and general-domain pre-trained DPR models on the BioSF dataset, achieving up to 19 points of improvement in hits@10 on a subset of PubMed.
- Reader: BioBERT models fine-tuned on BioSF achieve substantially higher performance (e.g., 54.80/55.65 Exact Match for BioBERT-large) compared to baselines trained on BioASQ or zero-shot RE datasets.
- End-to-End: The full slot-filling system achieves a micro-recall of 24.38% on the standard BioSF test set and 18.66% in a zero-shot setting using Hetionet. While these numbers might seem low, the authors attribute this to the inherent difficulty and imperfect evaluation settings.
Limitations and Future Work
- Evaluation challenges: The authors acknowledge that evaluating biomedical QA is difficult due to the potential for multiple valid answers that might not be present in their knowledge base. This leads to an "imperfect, i.e., 'silver' quality, evaluation regime."
- Future directions: The authors plan to explore sequence-to-sequence variants of their approach and conduct a more thorough comparison between their slot-filling system and traditional biomedical IE systems. They also aim to further improve the models and datasets.
Summary updated