Named Entity Recognition in Electronic Health Records Using Transfer Learning Bootstrapped Neural Networks

Luka Gligic, Andrey Kormilitzin, Paul Goldberg, Alejo J. Nevado-Holgado

Electronic Health Records (EHRs) are a goldmine for biomedical research, containing detailed patient histories, treatment outcomes, and diagnostic nuances. However, the most valuable information is often "locked" in unstructured clinical notes—text filled with idiosyncratic shorthand, non-standard acronyms, and complex medical jargon. While deep learning has revolutionized Natural Language Processing (NLP), these models typically demand massive labeled datasets that are notoriously difficult to obtain in medicine due to privacy concerns and the high cost of expert clinician time.

In this paper, researchers from the University of Oxford present a transfer learning approach that "bootstraps" neural networks using unannotated data, setting a new bar for performance in medical information extraction.

TL;DR

By pre-training word embeddings on a large pool of unannotated clinical text, the authors developed a neural architecture that achieved a 94.6 F1 score on the I2B2 2009 Medical Extraction Challenge—outperforming the original competition winner by 4.3 points. The study demonstrates that domain-specific transfer learning is the key to making deep learning viable for data-scarce medical applications.

What Problem Does This Paper Address?

The central challenge is data scarcity. Neural Networks (NNs) are powerful but data-hungry. In the clinical domain, we have millions of records (unannotated), but only a tiny fraction are labeled by experts for specific tasks like Named Entity Recognition (NER).

Furthermore, clinical text is "messy." A doctor might write “transport pt to OT tid via W/C” instead of “transport patient to occupational therapy three times a day via wheelchair.” Traditional rule-based systems struggle with this variability, while standard NNs often fail to generalize when trained on only a few hundred labeled examples.

Methods

The authors' core strategy is a two-stage transfer learning pipeline:

  1. Unsupervised Pre-training: They first trained word embeddings (using both CBOW and Continuous Skip-Gram algorithms) on a large pool of ~4,300 unannotated EHR documents. This allowed the model to learn the "language of medicine"—the semantic relationships between drugs, dosages, and symptoms—without needing any labels.
  2. Supervised Fine-tuning: These pre-trained embeddings served as the foundation for several downstream architectures:
  3. Context-free FNN: A simple feed-forward network looking at individual words.
  4. Context-aware FNN: An FNN that considers a window of surrounding words.
  5. RNN (LSTM): A recurrent model that sequentially processes the text to capture long-range dependencies.
  6. Seq2Seq with Attention: Used specifically for Relation Extraction (RE), identifying how medications relate to their specific dosages, frequencies, and reasons.
Architectures for term classification (NER)
Figure 1: The three architectures compared for NER: context-free FNN, context-aware FNN, and the LSTM-based RNN.

Datasets

The study utilized a collection of datasets released by the I2B2 (Informatics for Integrating Biology and the Bedside) challenges from 2007 to 2012.

  • Unannotated Pool: 4,347 unique documents used for training embeddings.
  • Annotated Set: 258 documents from the 2009 challenge, specifically labeled for medications and their attributes (dosage, mode, frequency, duration, and reason).

Key Insights and Ideas

  • Domain-Specific Embeddings are Essential: General-purpose embeddings (like those trained on Wikipedia) aren't enough for clinical notes. Pre-training on actual EHR text was the primary driver of performance gains.
  • RNNs Rule the Sequence: The LSTM-based RNN significantly outperformed the FNN models. Its ability to "remember" the sequence of a sentence is vital for distinguishing between, for example, a medication the patient is taking versus one they are allergic to.
  • Dropout for Small Data: The authors found that high dropout rates (a regularization technique) were particularly effective for the most sparsely annotated categories, preventing the model from overfitting to the few examples it had.
  • The "Reason" Gap: Interestingly, the models struggled most with identifying the reason for a prescription. While a human can easily infer a reason from context, the NN only achieved an F1 of 28.4 in this category, trailing behind traditional rule-based winners.

Key Results

  • NER Performance: The RNN architecture achieved a 94.6 F1 score on the I2B2 2009 task, a significant jump over the 90.3 F1 score of the original competition winner.
  • Relation Extraction: Using a seq2seq model with attention, they achieved an 82.4 F1 score for linking medications to their attributes, a critical step for epidemiological research.
  • Window Size: For the context-aware FNN, performance peaked with small windows, whereas the RNN was much more robust to changes in window size (see Figure 4).
Effects of window size on F1 score
Figure 4: F1 score variance based on window size for FNN (left) and RNN (right).

Limitations and Future Work

The most prominent limitation is the model's poor performance on the "reason" category. The authors note that a non-expert human can learn to recognize reasons for medication after seeing just a few examples, yet the NN remains relatively "blind" even after hundreds.

Future work may involve:

  • Few-shot learning: Developing models that learn more like humans from very few examples.
  • Hybrid Systems: Combining neural networks with symbolic methods (like medical dictionaries and gazetteers) to inject explicit domain knowledge.
  • Larger Pre-training: Leveraging even larger, multi-domain corpora to further refine the initial word embeddings.

Summary updated

Abstract

Neural networks (NNs) have become the state of the art in many machine learning applications, especially in image and sound processing [1]. The same, although to a lesser extent [2,3], could be said in natural language processing (NLP) tasks, such as named entity recognition. However, the success of NNs remains dependent on the availability of large labelled datasets, which is a significant hurdle in many important applications. One such case are electronic health records (EHRs), which are arguably the largest source of medical data, most of which lies hidden in natural text [4,5]. Data access is difficult due to data privacy concerns, and therefore annotated datasets are scarce. With scarce data, NNs will likely not be able to extract this hidden information with practical accuracy. In our study, we develop an approach that solves these problems for named entity recognition, obtaining 94.6 F1 score in I2B2 2009 Medical Extraction Challenge [6], 4.3 above the architecture that won the competition. Beyond the official I2B2 challenge, we further achieve 82.4 F1 on extracting relationships between medical terms. To reach this state-of-the-art accuracy, our approach applies transfer learning to leverage on datasets annotated for other I2B2 tasks, and designs and trains embeddings that specially benefit from such transfer.
BibTeX
@article{gligic2020named,
  author = {Luka Gligic and Andrey Kormilitzin and Paul Goldberg and Alejo J. Nevado-Holgado},
  title = {{Named Entity Recognition in Electronic Health Records Using Transfer Learning Bootstrapped Neural Networks}},
  journal = {Neural Networks},
  year = {2020},
  doi = {10.1016/j.neunet.2019.08.032},
}
Figure