TL;DR
This paper tackles the challenge of noisy data in distantly supervised biomedical relation extraction by introducing a novel entity marking strategy called "k-tag." This approach significantly reduces noise and achieves state-of-the-art performance by leveraging the order of entities as they appear in a knowledge base, simplifying previous complex methods.
What Problem Does This Paper Address?
Extracting relationships between entities in biomedical texts is crucial for advancing research. Traditional supervised methods require extensive manual annotation, which is costly and time-consuming, especially in specialized domains like biomedicine. Distant supervision offers a solution by automatically generating labeled data from existing knowledge bases (KBs). However, this process introduces significant noise, as sentences linked to a KB fact might not actually express that specific relation. This noise can degrade model performance. Existing methods have tried to mitigate this noise through various techniques, but this paper focuses on a more direct and simpler approach by improving how entities are represented within the model.
Methods
The core contribution of this paper is the k-tag (knowledge base ordered) entity marking strategy. Unlike previous methods that mark entities based on their order in the sentence (s-tag), k-tag marks entities based on their order as defined in the knowledge base. This is achieved by adding special tokens around the head and tail entities in a sentence, irrespective of their order in the text. This simple encoding directly captures the latent direction of the relation as it's represented in the KB, thereby reducing noise.
The authors extend a BERT-based relation extraction model (R-BERT) to a multiple instance learning (MIL) framework to handle bags of sentences associated with a KB fact. They compare their k-tag approach with the s-tag approach, using both average (avg) and selective attention (attn) bag aggregation functions.
Datasets
The study uses the UMLS (Unified Medical Language System) as the knowledge base and MEDLINE abstracts as the text source. The data creation pipeline is described in detail, including processing UMLS concepts and MEDLINE abstracts to generate triples and sentences. The authors created three different data splits to support their experimental setups, ensuring no overlapping facts across train, validation, and test sets.
Key Insights and Ideas
- KB Order Matters: The order of entities in the knowledge base is a surprisingly powerful signal for reducing noise in distantly supervised relation extraction. It directly encodes the intended direction of the relation.
- Simplicity Wins: The k-tag approach, by directly leveraging KB information, significantly simplifies previous methods that relied on more complex techniques like joint modeling with knowledge graph completion (KGC) or additional auxiliary tasks.
- Attention Isn't Always Best: For distantly supervised settings with potentially undersized bags (few sentences), simple averaging can outperform attention mechanisms, which struggle to learn meaningful distributions when faced with limited and noisy data.
- Data Quality is Paramount: The paper emphasizes that a well-designed data pipeline and effective noise reduction strategies are crucial for achieving high performance in distantly supervised relation extraction.
Key Results
The models employing the k-tag strategy consistently outperformed those using the s-tag strategy across various metrics. Specifically, k-tag with average aggregation achieved state-of-the-art Precision@k (P@2k, P@4k, P@6k) compared to previous best results. For instance, the k-tag + avg model achieved a Precision@2k of 0.983, significantly higher than the previous best of 0.829. The PR curves also visually demonstrate the superiority of k-tag.
Limitations and Future Work
The authors acknowledge that their work focuses on a specific type of noise (sentence-level noise) and that other sources of noise might exist. They also note that while their method simplifies previous approaches, it still relies on pre-trained language models like BERT. Future work could explore applying this k-tag strategy to other types of relation extraction tasks or investigate more sophisticated ways to handle the inherent noise in distantly supervised data. The authors also make their code and data creation pipeline publicly available, encouraging further research.
Summary updated