NCUEE at MEDIQA 2019: Medical Text Inference Using Ensemble BERT-BiLSTM-Attention Model

Lung-Hao Lee, Yi Lu, Po-Han Chen, Po-Lei Lee, Kuo-Kai ShyuACL 2019

TL;DR

This paper presents the NCUEE system for the MEDIQA 2019 challenge, tackling medical text inference. They achieved a strong third-place ranking by combining BERT embeddings with a BiLSTM network and an attention mechanism, demonstrating the power of pre-trained language models for specialized medical NLP tasks.

What Problem Does This Paper Address?

The paper addresses the challenge of Natural Language Inference (NLI) specifically within the medical domain. NLI involves determining the relationship between two sentences: whether one entails, contradicts, or is neutral with respect to the other. While general NLI datasets like SNLI exist, they are often limited to a single text genre. The MedNLI dataset, used here, is derived from patient medical histories, making it crucial for real-world medical applications. The MEDIQA 2019 challenge specifically focused on NLI, Recognizing Question Entailment (RQE), and Question Answering (QA) in this medical context, highlighting the need for robust models that can understand complex medical language.

Methods

The NCUEE system employs a BERT-BiLSTM-Attention architecture.

  • BERT Embedding: They leverage BERT (Bidirectional Encoder Representations from Transformers) as a powerful word embedding method. This means they use pre-trained BERT models to convert words into dense vector representations that capture rich contextual information.
  • BiLSTM Network: The BERT embeddings are then fed into a Bidirectional Long Short-Term Memory (BiLSTM) network. BiLSTMs are well-suited for sequential data like text, as they can process information from both forward and backward directions, capturing dependencies across the entire sentence.
  • Attention Mechanism: An attention layer is integrated to allow the model to focus on the most relevant words in the sentence pair when making an inference. This helps the model weigh the importance of different words, leading to more accurate predictions.
  • Ensemble Strategy: For the final submission, they used a majority voting ensemble strategy, training their best model multiple times and combining their predictions to improve robustness and accuracy.

Datasets

  • MedNLI: The primary dataset used for training, validation, and testing is the MedNLI dataset. This dataset is specifically curated from patient medical histories, making it relevant for medical text inference tasks.
  • Training/Validation/Test Split: The authors used 12,627 sentence-pairs for training and 1,422 for validation during system development. The final evaluation was performed on a test set of 405 sentence-pairs.

Key Insights and Ideas

  • BERT's Efficacy in Medical Domain: The paper strongly suggests that pre-trained language models like BERT, and specifically BioBERT (a BERT variant trained on biomedical corpora), are highly effective for medical text inference.
  • BioBERT Advantage: Fine-tuning BioBERT models consistently outperformed their general BERT counterparts, underscoring the benefit of domain-specific pre-training.
  • Attention Enhances BiLSTM: The integration of an attention mechanism with BiLSTM significantly improved performance, allowing the model to better discern important contextual cues for inference.
  • Ensembling for Robustness: A simple majority voting ensemble strategy, even with a single model trained multiple times, provided a performance boost, highlighting the value of ensemble methods.
  • Dataset Size and Potential for Manipulation: The authors note that the small test set size (405 instances) and its structured format (one premise with three hypotheses) made it susceptible to post-editing rules, which could artificially inflate scores.

Key Results

  • The NCUEE system achieved an accuracy score of 0.84 on the MEDIQA 2019 NLI task.
  • This score ranked them third among all 42 participating teams on the leaderboard.
  • After excluding invalid submissions, their score of 0.84 ranked them 11th out of 17 valid submissions.
  • The BioBERT-Base + BiLSTM-Attention model achieved a higher accuracy (0.805) than the BERT-Large + BiLSTM-Attention model (0.808) in one experiment, and the ensemble model achieved the highest accuracy of 0.84.
  • A post-editing rule, applied to the test set, could artificially boost the best model's accuracy to 0.975.

Limitations and Future Work

  • Small Test Set: The authors acknowledge the extremely small size of the test set (405 instances) and its structured pattern, which could allow for manipulation and might not fully represent real-world performance.
  • Human Intervention: The relatively long testing period and the structured test set suggest that human intervention could have influenced results.
  • Future Work: The authors suggest exploring other textual features to further improve performance, indicating that their current model could be enhanced by incorporating additional linguistic or domain-specific information.

Summary updated

BibTeX
@inproceedings{lee2019ncuee,
  author = {Lung-Hao Lee and Yi Lu and Po-Han Chen and Po-Lei Lee and Kuo-Kai Shyu},
  title = {{NCUEE at MEDIQA 2019: Medical Text Inference Using Ensemble BERT-BiLSTM-Attention Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure