TL;DR
This paper reports on the IITP team's participation in the MEDIQA 2019 shared task, focusing on natural language inference, question entailment, and question answering within the biomedical domain. They achieved strong results by leveraging pre-trained transformer models like BERT and BioBERT, demonstrating their effectiveness in these specialized NLP tasks.
What Problem Does This Paper Address?
The biomedical domain presents unique challenges for Natural Language Processing (NLP) due to its specialized terminology and complex relationships between medical concepts. The MEDIQA 2019 shared task aimed to advance research in three key areas within this domain:
- Natural Language Inference (NLI): Determining the relationship (entailment, neutral, or contradiction) between two medical text snippets. This is crucial for tasks like summarizing medical literature or understanding patient records.
- Recognizing Question Entailment (RQE): Identifying if the answer to one medical question is also a complete or partial answer to another. This is vital for building more intelligent medical question-answering systems that can handle nuanced queries.
- Question Answering (QA): Filtering and improving the ranking of automatically retrieved answers to medical questions. This directly impacts the usability and accuracy of medical information retrieval systems.
The paper addresses the need for robust NLP systems capable of handling these specific medical language tasks, building upon recent advancements in deep learning.
Methods
The IITP team employed a variety of deep learning approaches, primarily centered around transformer-based models:
- BERT and BioBERT: These models, pre-trained on vast amounts of text (BERT on general text, BioBERT on biomedical literature like PubMed abstracts and PMC full-text articles), served as the foundation for most of their systems. BioBERT, in particular, showed an advantage due to its domain-specific pre-training.
- Transformer Architectures: For NLI and RQE, they extracted the "[CLS]" token's final hidden state from BERT/BioBERT as a sentence-pair representation. This was then fed into feed-forward layers for classification.
- Siamese Networks with Bi-LSTMs: For RQE, they also explored Siamese architectures using Bidirectional Long Short-Term Memory (Bi-LSTM) networks to learn sentence similarity between question pairs.
- Ensembling: For both NLI and QA, they combined predictions from multiple trained models (ensembling) to improve robustness and accuracy.
- BM25: In the QA task, they combined BERT/BioBERT's predicted relevance scores with the traditional BM25 algorithm for ranking answers.
They experimented with different fine-tuning strategies, including training only the feed-forward layers or fine-tuning the entire model, and explored the impact of using different pre-training datasets (e.g., PubMed abstracts only vs. PubMed + PMC).
Datasets
The experiments were conducted using datasets provided by the MEDIQA 2019 shared task organizers.
- NLI Task: The dataset consisted of training instances with 14,049 sentence pairs and test instances with 405 sentence pairs.
- RQE Task: The training set had 8,588 pairs, with a validation set of 302 pairs and a test set of 230 pairs.
- QA Task: The training data was split into two domains (LiveQAMed and Alexa), each with 104 questions. The validation set had 25 questions, and the test set contained 150 question pairs.
The paper also mentions using the MedNLI dataset for training in one of their NLI runs.
Key Insights and Ideas
- BioBERT's Domain Advantage: BioBERT consistently outperformed the general BERT model, highlighting the significant benefit of pre-training on domain-specific biomedical text for tasks like NLI and RQE.
- Fine-tuning vs. Feature Extraction: For RQE, fine-tuning BERT seemed to hurt performance, while using pre-trained BERT embeddings without fine-tuning yielded better results. This suggests that for certain tasks, the pre-trained representations are already highly effective.
- Ensembling Boosts Performance: Combining predictions from multiple models (especially ensembles of BioBERT models) led to the highest accuracies in both NLI and QA, demonstrating the power of model diversity.
- Siamese Networks for RQE: Siamese architectures with Bi-LSTMs showed promising results for the RQE task, indicating their suitability for learning similarity between question pairs.
- BERT + BM25 Synergy: The combination of BERT's learned relevance scores with the established BM25 ranking algorithm proved effective for the QA task.
Key Results
The IITP team achieved competitive results across all three tasks:
- NLI: Their best run achieved an accuracy of 81.8%, placing them 12th out of 17 teams.
- RQE: Their top performance reached 53.2% accuracy, securing them 10th position among 12 teams.
- QA: Their best run achieved an accuracy of 71.7%, ranking them 6th out of 10 teams.
They also reported other metrics like Spearman's Rho, MRR, and Precision for the QA task, with their best run achieving an MRR of 0.8611 and Precision of 0.7936.
Limitations and Future Work
- Limitations: The paper notes that fine-tuning BERT with a batch size of 32 and a maximum sequence length of 256 is memory-intensive, leading them to use smaller batch sizes for the QA task. They also observed that for RQE, fine-tuning BERT sometimes led to decreased performance.
- Future Work: The authors suggest several avenues for future research:
- Conducting a detailed analysis of their top-scoring models to understand their underlying techniques.
- Exploring Embedding from Language Model (EMLo) based models for NLI and comparing them with BERT-based approaches.
Summary updated